Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250626 19:57:10.495054 4282 test_util.cc:276] Using random seed: -141771538
2025-06-26T19:57:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T19:57:10Z Disabled control of system clock
I20250626 19:57:10.562889 4282 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
/tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.190:33823
--webserver_interface=127.4.46.190
--webserver_port=0
--builtin_ntp_servers=127.4.46.148:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.46.190:33823,127.4.46.189:45039,127.4.46.188:40801
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250626 19:57:10.855919 4291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 19:57:10.856498 4291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 19:57:10.856909 4291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 19:57:10.891902 4291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 19:57:10.892199 4291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 19:57:10.892400 4291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 19:57:10.892592 4291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 19:57:10.926488 4291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.46.148:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.46.190:33823,127.4.46.189:45039,127.4.46.188:40801
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.190:33823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.46.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6785
TSAN enabled
I20250626 19:57:10.927806 4291 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 19:57:10.929352 4291 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 19:57:10.945523 4300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:57:10.946707 4297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:57:10.945556 4298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:57:12.112972 4299 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 19:57:12.113076 4291 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 19:57:12.117511 4291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 19:57:12.121069 4291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 19:57:12.122669 4291 hybrid_clock.cc:648] HybridClock initialized: now 1750967832122603 us; error 68 us; skew 500 ppm
I20250626 19:57:12.123811 4291 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 19:57:12.132436 4291 webserver.cc:469] Webserver started at http://127.4.46.190:36587/ using document root <none> and password file <none>
I20250626 19:57:12.133576 4291 fs_manager.cc:362] Metadata directory not provided
I20250626 19:57:12.133841 4291 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 19:57:12.134488 4291 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 19:57:12.140530 4291 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data/instance:
uuid: "e6c58006b6be41c1bd6bdb5eb010d360"
format_stamp: "Formatted at 2025-06-26 19:57:12 on dist-test-slave-48dm"
I20250626 19:57:12.141902 4291 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/wal/instance:
uuid: "e6c58006b6be41c1bd6bdb5eb010d360"
format_stamp: "Formatted at 2025-06-26 19:57:12 on dist-test-slave-48dm"
I20250626 19:57:12.152587 4291 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.012s sys 0.000s
I20250626 19:57:12.161510 4307 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 19:57:12.162739 4291 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20250626 19:57:12.163146 4291 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data,/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/wal
uuid: "e6c58006b6be41c1bd6bdb5eb010d360"
format_stamp: "Formatted at 2025-06-26 19:57:12 on dist-test-slave-48dm"
I20250626 19:57:12.163566 4291 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 19:57:12.220364 4291 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 19:57:12.221858 4291 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 19:57:12.222316 4291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 19:57:12.300675 4291 rpc_server.cc:307] RPC server started. Bound to: 127.4.46.190:33823
I20250626 19:57:12.300755 4358 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.46.190:33823 every 8 connection(s)
I20250626 19:57:12.303268 4291 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data/info.pb
I20250626 19:57:12.307590 4282 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu as pid 4291
I20250626 19:57:12.308121 4282 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/wal/instance
I20250626 19:57:12.309656 4359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:12.313448 4282 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
/tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.189:45039
--webserver_interface=127.4.46.189
--webserver_port=0
--builtin_ntp_servers=127.4.46.148:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.46.190:33823,127.4.46.189:45039,127.4.46.188:40801
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250626 19:57:12.322955 4359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } has no permanent_uuid. Determining permanent_uuid...
I20250626 19:57:12.350339 4359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } has no permanent_uuid. Determining permanent_uuid...
W20250626 19:57:12.352919 4310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.46.189:45039: connect: Connection refused (error 111)
W20250626 19:57:12.357419 4359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.189:45039: Network error: Client connection negotiation failed: client connection to 127.4.46.189:45039: connect: Connection refused (error 111)
I20250626 19:57:12.406889 4359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } attempt: 1
W20250626 19:57:12.411180 4359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.189:45039: Network error: Client connection negotiation failed: client connection to 127.4.46.189:45039: connect: Connection refused (error 111)
I20250626 19:57:12.479555 4359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } attempt: 2
W20250626 19:57:12.483891 4359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.189:45039: Network error: Client connection negotiation failed: client connection to 127.4.46.189:45039: connect: Connection refused (error 111)
I20250626 19:57:12.575278 4359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } attempt: 3
W20250626 19:57:12.579367 4359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.189:45039: Network error: Client connection negotiation failed: client connection to 127.4.46.189:45039: connect: Connection refused (error 111)
W20250626 19:57:12.618525 4361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 19:57:12.619161 4361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 19:57:12.619616 4361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 19:57:12.649616 4361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 19:57:12.649950 4361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 19:57:12.650205 4361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 19:57:12.650454 4361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 19:57:12.684544 4361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.46.148:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.46.190:33823,127.4.46.189:45039,127.4.46.188:40801
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.189:45039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.46.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6785
TSAN enabled
I20250626 19:57:12.685899 4361 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 19:57:12.687542 4361 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 19:57:12.703441 4370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:57:12.722800 4359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } attempt: 4
W20250626 19:57:12.726953 4359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.189:45039: Network error: Client connection negotiation failed: client connection to 127.4.46.189:45039: connect: Connection refused (error 111)
I20250626 19:57:13.026319 4359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } attempt: 5
W20250626 19:57:13.030025 4359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.189:45039: Network error: Client connection negotiation failed: client connection to 127.4.46.189:45039: connect: Connection refused (error 111)
I20250626 19:57:13.577705 4359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } attempt: 6
W20250626 19:57:13.587330 4359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.189:45039: Network error: Client connection negotiation failed: client connection to 127.4.46.189:45039: connect: Connection refused (error 111)
W20250626 19:57:12.703583 4371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:57:12.703610 4373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:57:12.705169 4361 server_base.cc:1048] running on GCE node
I20250626 19:57:13.898254 4361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 19:57:13.900777 4361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 19:57:13.902122 4361 hybrid_clock.cc:648] HybridClock initialized: now 1750967833902064 us; error 67 us; skew 500 ppm
I20250626 19:57:13.902972 4361 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 19:57:13.910215 4361 webserver.cc:469] Webserver started at http://127.4.46.189:34749/ using document root <none> and password file <none>
I20250626 19:57:13.911168 4361 fs_manager.cc:362] Metadata directory not provided
I20250626 19:57:13.911396 4361 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 19:57:13.911828 4361 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 19:57:13.915874 4361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/data/instance:
uuid: "62ac80df82774b9b87c0be076b5637af"
format_stamp: "Formatted at 2025-06-26 19:57:13 on dist-test-slave-48dm"
I20250626 19:57:13.916957 4361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/wal/instance:
uuid: "62ac80df82774b9b87c0be076b5637af"
format_stamp: "Formatted at 2025-06-26 19:57:13 on dist-test-slave-48dm"
I20250626 19:57:13.923740 4361 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250626 19:57:13.929278 4382 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 19:57:13.930226 4361 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250626 19:57:13.930532 4361 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/data,/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/wal
uuid: "62ac80df82774b9b87c0be076b5637af"
format_stamp: "Formatted at 2025-06-26 19:57:13 on dist-test-slave-48dm"
I20250626 19:57:13.930891 4361 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 19:57:13.977631 4361 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 19:57:13.979060 4361 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 19:57:13.979511 4361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 19:57:14.054159 4361 rpc_server.cc:307] RPC server started. Bound to: 127.4.46.189:45039
I20250626 19:57:14.054227 4433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.46.189:45039 every 8 connection(s)
I20250626 19:57:14.056805 4361 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/data/info.pb
I20250626 19:57:14.059723 4282 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu as pid 4361
I20250626 19:57:14.060284 4282 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-1/wal/instance
I20250626 19:57:14.062906 4434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:14.063683 4282 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
/tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.188:40801
--webserver_interface=127.4.46.188
--webserver_port=0
--builtin_ntp_servers=127.4.46.148:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.46.190:33823,127.4.46.189:45039,127.4.46.188:40801
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250626 19:57:14.073513 4434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } has no permanent_uuid. Determining permanent_uuid...
I20250626 19:57:14.094069 4434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } has no permanent_uuid. Determining permanent_uuid...
I20250626 19:57:14.107184 4434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } has no permanent_uuid. Determining permanent_uuid...
W20250626 19:57:14.110010 4383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.46.188:40801: connect: Connection refused (error 111)
W20250626 19:57:14.114683 4434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:40801: Network error: Client connection negotiation failed: client connection to 127.4.46.188:40801: connect: Connection refused (error 111)
I20250626 19:57:14.164147 4434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } attempt: 1
W20250626 19:57:14.168345 4434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:40801: Network error: Client connection negotiation failed: client connection to 127.4.46.188:40801: connect: Connection refused (error 111)
I20250626 19:57:14.236807 4434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } attempt: 2
W20250626 19:57:14.241007 4434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:40801: Network error: Client connection negotiation failed: client connection to 127.4.46.188:40801: connect: Connection refused (error 111)
I20250626 19:57:14.332489 4434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } attempt: 3
W20250626 19:57:14.336884 4434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:40801: Network error: Client connection negotiation failed: client connection to 127.4.46.188:40801: connect: Connection refused (error 111)
W20250626 19:57:14.371428 4436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 19:57:14.371944 4436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 19:57:14.372327 4436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 19:57:14.402669 4436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 19:57:14.402957 4436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 19:57:14.403153 4436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 19:57:14.403337 4436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 19:57:14.437508 4436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.46.148:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.46.190:33823,127.4.46.189:45039,127.4.46.188:40801
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.188:40801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.46.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6785
TSAN enabled
I20250626 19:57:14.438802 4436 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 19:57:14.440375 4436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 19:57:14.456881 4449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:57:14.480376 4434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } attempt: 4
W20250626 19:57:14.483876 4434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:40801: Network error: Client connection negotiation failed: client connection to 127.4.46.188:40801: connect: Connection refused (error 111)
I20250626 19:57:14.647987 4359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } attempt: 7
I20250626 19:57:14.661065 4359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } has no permanent_uuid. Determining permanent_uuid...
W20250626 19:57:14.667472 4359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:40801: Network error: Client connection negotiation failed: client connection to 127.4.46.188:40801: connect: Connection refused (error 111)
I20250626 19:57:14.725823 4359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } attempt: 1
W20250626 19:57:14.730157 4359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:40801: Network error: Client connection negotiation failed: client connection to 127.4.46.188:40801: connect: Connection refused (error 111)
I20250626 19:57:14.783484 4434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } attempt: 5
W20250626 19:57:14.788079 4434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:40801: Network error: Client connection negotiation failed: client connection to 127.4.46.188:40801: connect: Connection refused (error 111)
I20250626 19:57:14.811558 4359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } attempt: 2
W20250626 19:57:14.815914 4359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:40801: Network error: Client connection negotiation failed: client connection to 127.4.46.188:40801: connect: Connection refused (error 111)
I20250626 19:57:14.901283 4359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } attempt: 3
W20250626 19:57:14.906045 4359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:40801: Network error: Client connection negotiation failed: client connection to 127.4.46.188:40801: connect: Connection refused (error 111)
I20250626 19:57:15.046437 4359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } attempt: 4
W20250626 19:57:15.051126 4359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:40801: Network error: Client connection negotiation failed: client connection to 127.4.46.188:40801: connect: Connection refused (error 111)
I20250626 19:57:15.334554 4359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } attempt: 5
I20250626 19:57:15.335672 4434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } attempt: 6
W20250626 19:57:15.340269 4359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:40801: Network error: Client connection negotiation failed: client connection to 127.4.46.188:40801: connect: Connection refused (error 111)
W20250626 19:57:15.342897 4434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:40801: Network error: Client connection negotiation failed: client connection to 127.4.46.188:40801: connect: Connection refused (error 111)
W20250626 19:57:14.457181 4446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:57:14.458904 4447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:57:14.460404 4436 server_base.cc:1048] running on GCE node
I20250626 19:57:15.652042 4436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 19:57:15.655185 4436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 19:57:15.656632 4436 hybrid_clock.cc:648] HybridClock initialized: now 1750967835656593 us; error 58 us; skew 500 ppm
I20250626 19:57:15.657425 4436 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 19:57:15.664490 4436 webserver.cc:469] Webserver started at http://127.4.46.188:44227/ using document root <none> and password file <none>
I20250626 19:57:15.665380 4436 fs_manager.cc:362] Metadata directory not provided
I20250626 19:57:15.665562 4436 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 19:57:15.665971 4436 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 19:57:15.670290 4436 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/data/instance:
uuid: "ae949d1fcdf54e87930871ac0ae7da31"
format_stamp: "Formatted at 2025-06-26 19:57:15 on dist-test-slave-48dm"
I20250626 19:57:15.671360 4436 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/wal/instance:
uuid: "ae949d1fcdf54e87930871ac0ae7da31"
format_stamp: "Formatted at 2025-06-26 19:57:15 on dist-test-slave-48dm"
I20250626 19:57:15.678649 4436 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250626 19:57:15.684044 4461 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 19:57:15.685029 4436 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250626 19:57:15.685312 4436 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/data,/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/wal
uuid: "ae949d1fcdf54e87930871ac0ae7da31"
format_stamp: "Formatted at 2025-06-26 19:57:15 on dist-test-slave-48dm"
I20250626 19:57:15.685602 4436 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 19:57:15.737071 4436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 19:57:15.738435 4436 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 19:57:15.738852 4436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 19:57:15.818009 4436 rpc_server.cc:307] RPC server started. Bound to: 127.4.46.188:40801
I20250626 19:57:15.818104 4512 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.46.188:40801 every 8 connection(s)
I20250626 19:57:15.820581 4436 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/data/info.pb
I20250626 19:57:15.825814 4513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:15.827140 4282 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu as pid 4436
I20250626 19:57:15.827527 4282 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-2/wal/instance
I20250626 19:57:15.838670 4513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } has no permanent_uuid. Determining permanent_uuid...
I20250626 19:57:15.870924 4513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } has no permanent_uuid. Determining permanent_uuid...
I20250626 19:57:15.887233 4513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } has no permanent_uuid. Determining permanent_uuid...
I20250626 19:57:15.894212 4359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } attempt: 6
I20250626 19:57:15.917357 4513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31: Bootstrap starting.
I20250626 19:57:15.919765 4359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Bootstrap starting.
I20250626 19:57:15.926227 4513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:15.926307 4359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:15.928570 4513 log.cc:826] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31: Log is configured to *not* fsync() on all Append() calls
I20250626 19:57:15.928791 4359 log.cc:826] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Log is configured to *not* fsync() on all Append() calls
I20250626 19:57:15.933698 4359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: No bootstrap required, opened a new log
I20250626 19:57:15.933713 4513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31: No bootstrap required, opened a new log
I20250626 19:57:15.953579 4513 raft_consensus.cc:357] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:15.953601 4359 raft_consensus.cc:357] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:15.954217 4513 raft_consensus.cc:383] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:15.954182 4359 raft_consensus.cc:383] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:15.954435 4513 raft_consensus.cc:738] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae949d1fcdf54e87930871ac0ae7da31, State: Initialized, Role: FOLLOWER
I20250626 19:57:15.954425 4359 raft_consensus.cc:738] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c58006b6be41c1bd6bdb5eb010d360, State: Initialized, Role: FOLLOWER
I20250626 19:57:15.955170 4359 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:15.955163 4513 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:15.959065 4526 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:15.960207 4359 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 19:57:15.960477 4526 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:15.962177 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:15.963582 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:15.965593 4513 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: configured and running, proceeding with master startup.
W20250626 19:57:15.981585 4313 tablet.cc:2378] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 19:57:15.994421 4467 tablet.cc:2378] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 19:57:16.001367 4547 catalog_manager.cc:1560] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:57:16.001789 4547 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:57:16.015061 4550 catalog_manager.cc:1560] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:57:16.016084 4550 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:57:15.996165 4291 master_runner.cc:186] Error getting master registration for 127.4.46.189:45039: OK, instance_id { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" instance_seqno: 1750967834034461 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250626 19:57:16.034765 4291 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 19:57:16.013336 4436 master_runner.cc:186] Error getting master registration for 127.4.46.189:45039: OK, instance_id { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" instance_seqno: 1750967834034461 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250626 19:57:16.054064 4436 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 19:57:16.136828 4526 raft_consensus.cc:491] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:16.137400 4526 raft_consensus.cc:513] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:16.139973 4526 leader_election.cc:290] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:16.141940 4488 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31" is_pre_election: true
I20250626 19:57:16.142836 4488 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6c58006b6be41c1bd6bdb5eb010d360 in term 0.
I20250626 19:57:16.144055 4308 leader_election.cc:304] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31, e6c58006b6be41c1bd6bdb5eb010d360; no voters:
I20250626 19:57:16.144838 4526 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:16.145184 4526 raft_consensus.cc:491] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:16.145504 4526 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:16.153163 4526 raft_consensus.cc:513] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:16.155717 4526 leader_election.cc:290] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 1 election: Requested vote from peers 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:16.161096 4488 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31"
I20250626 19:57:16.161652 4488 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:16.177764 4488 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6c58006b6be41c1bd6bdb5eb010d360 in term 1.
I20250626 19:57:16.181483 4308 leader_election.cc:304] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31, e6c58006b6be41c1bd6bdb5eb010d360; no voters:
I20250626 19:57:16.182304 4526 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:16.190052 4526 raft_consensus.cc:695] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 1 LEADER]: Becoming Leader. State: Replica: e6c58006b6be41c1bd6bdb5eb010d360, State: Running, Role: LEADER
I20250626 19:57:16.191200 4526 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:16.209590 4552 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6c58006b6be41c1bd6bdb5eb010d360. Latest consensus state: current_term: 1 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:16.211719 4552 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: This master's current role is: LEADER
I20250626 19:57:16.220616 4554 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 19:57:16.227576 4554 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 19:57:16.254875 4488 raft_consensus.cc:1273] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Refusing update from remote peer e6c58006b6be41c1bd6bdb5eb010d360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:57:16.256634 4552 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:16.291209 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6c58006b6be41c1bd6bdb5eb010d360. Latest consensus state: current_term: 1 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:16.293108 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:16.295127 4555 mvcc.cc:204] Tried to move back new op lower bound from 7171964257275953152 to 7171964257049915392. Current Snapshot: MvccSnapshot[applied={T|T < 7171964257275953152}]
I20250626 19:57:16.304181 4552 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:16.304893 4552 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: This master's current role is: LEADER
I20250626 19:57:16.309671 4554 catalog_manager.cc:1349] Generated new cluster ID: 845ffa7fa66f40c8abadc28aed775425
I20250626 19:57:16.309962 4554 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 19:57:16.313915 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:16.314563 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:16.370676 4554 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 19:57:16.373127 4554 catalog_manager.cc:1506] Loading token signing keys...
I20250626 19:57:16.403446 4434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } attempt: 7
I20250626 19:57:16.412019 4554 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Generated new TSK 0
I20250626 19:57:16.413054 4554 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 19:57:16.439293 4434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Bootstrap starting.
I20250626 19:57:16.448712 4434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:16.451336 4434 log.cc:826] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Log is configured to *not* fsync() on all Append() calls
I20250626 19:57:16.480463 4282 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
/tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.46.129:0
--local_ip_for_outbound_sockets=127.4.46.129
--webserver_interface=127.4.46.129
--webserver_port=0
--tserver_master_addrs=127.4.46.190:33823,127.4.46.189:45039,127.4.46.188:40801
--builtin_ntp_servers=127.4.46.148:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250626 19:57:16.507777 4434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: No bootstrap required, opened a new log
I20250626 19:57:16.580500 4434 raft_consensus.cc:357] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:16.582538 4434 raft_consensus.cc:383] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:16.583323 4434 raft_consensus.cc:738] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ac80df82774b9b87c0be076b5637af, State: Initialized, Role: FOLLOWER
I20250626 19:57:16.585584 4434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:16.606113 4409 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:16.607787 4408 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af"
I20250626 19:57:16.610440 4409 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6c58006b6be41c1bd6bdb5eb010d360 in term 0.
I20250626 19:57:16.614075 4407 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:16.619664 4409 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35872 (request call id 2) took 479 ms (client timeout 610 ms). Trace:
W20250626 19:57:16.620150 4409 rpcz_store.cc:269] 0626 19:57:16.140499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:16.140653 (+ 154us) service_pool.cc:225] Handling call
0626 19:57:16.613632 (+472979us) spinlock_profiling.cc:243] Waited 3.53 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72b481 00007fe78f7aaeda 00007fe790790057 00007fe78ef3bcc4 00007fe78ef3bc52 00007fe78ef3bb81 00007fe78ef3bafc 00007fe78ef3adb2 00007fe78d56fdcc 00007fe78d56f206 00007fe78d56ebaf
0626 19:57:16.619578 (+ 5946us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8110848}
I20250626 19:57:16.625247 4407 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Refusing update from remote peer e6c58006b6be41c1bd6bdb5eb010d360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:57:16.625880 4434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: configured and running, proceeding with master startup.
I20250626 19:57:16.630080 4552 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:16.626276 4563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:16.630914 4563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:16.683028 4566 mvcc.cc:204] Tried to move back new op lower bound from 7171964257275953152 to 7171964257049915392. Current Snapshot: MvccSnapshot[applied={T|T < 7171964257275953152 or (T in {7171964257275953152})}]
I20250626 19:57:16.701499 4563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6c58006b6be41c1bd6bdb5eb010d360. Latest consensus state: current_term: 1 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:16.701642 4564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:16.702325 4563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:16.702419 4564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:16.747155 4579 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250626 19:57:16.941898 4580 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:16.942644 4580 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:16.947171 4409 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:16.947167 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:16.948848 4580 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:16.948949 4462 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31; no voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:16.950349 4580 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 19:57:16.991676 4562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 19:57:16.992303 4562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 19:57:16.993227 4562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250626 19:57:17.021490 4550 catalog_manager.cc:1261] Loaded cluster ID: 845ffa7fa66f40c8abadc28aed775425
I20250626 19:57:17.022015 4550 catalog_manager.cc:1554] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31: loading cluster ID for follower catalog manager: success
I20250626 19:57:17.029639 4550 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31: acquiring CA information for follower catalog manager: success
W20250626 19:57:17.034209 4562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20250626 19:57:17.035250 4550 catalog_manager.cc:1604] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250626 19:57:17.035887 4562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.46.129
I20250626 19:57:17.072162 4562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.46.148:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.46.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.46.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.46.190:33823,127.4.46.189:45039,127.4.46.188:40801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.46.129
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6785
TSAN enabled
I20250626 19:57:17.073817 4562 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 19:57:17.076170 4562 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 19:57:17.097694 4586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:57:17.140774 4564 raft_consensus.cc:491] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:17.141539 4564 raft_consensus.cc:513] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:17.146584 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ac80df82774b9b87c0be076b5637af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:17.149428 4564 leader_election.cc:290] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:17.151289 4488 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ac80df82774b9b87c0be076b5637af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31" is_pre_election: true
I20250626 19:57:17.153435 4383 leader_election.cc:304] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ac80df82774b9b87c0be076b5637af; no voters: ae949d1fcdf54e87930871ac0ae7da31, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:17.155122 4564 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:57:17.255039 4552 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:17.255759 4552 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: This master's current role is: LEADER
W20250626 19:57:17.492025 4430 debug-util.cc:398] Leaking SignalData structure 0x7b080006f1a0 after lost signal to thread 4365
W20250626 19:57:17.492666 4430 debug-util.cc:398] Leaking SignalData structure 0x7b080006f1c0 after lost signal to thread 4433
I20250626 19:57:17.516952 4593 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:17.517482 4593 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:17.521216 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:17.524371 4409 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:17.525163 4593 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:17.527506 4464 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31; no voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:17.528765 4593 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:57:17.792513 4579 catalog_manager.cc:1261] Loaded cluster ID: 845ffa7fa66f40c8abadc28aed775425
I20250626 19:57:17.802943 4579 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: loading cluster ID for follower catalog manager: success
I20250626 19:57:17.862711 4579 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: acquiring CA information for follower catalog manager: success
I20250626 19:57:17.889901 4579 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 19:57:18.165169 4595 raft_consensus.cc:491] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:18.166189 4595 raft_consensus.cc:513] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:18.173574 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ac80df82774b9b87c0be076b5637af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:18.180415 4488 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ac80df82774b9b87c0be076b5637af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31" is_pre_election: true
I20250626 19:57:18.182379 4383 leader_election.cc:304] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ac80df82774b9b87c0be076b5637af; no voters: ae949d1fcdf54e87930871ac0ae7da31, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:18.195387 4596 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:57:18.196071 4595 leader_election.cc:290] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:18.727494 4597 raft_consensus.cc:491] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:18.728120 4597 raft_consensus.cc:513] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:18.731060 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ac80df82774b9b87c0be076b5637af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:18.735945 4488 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ac80df82774b9b87c0be076b5637af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31" is_pre_election: true
I20250626 19:57:18.736510 4597 leader_election.cc:290] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:18.738547 4383 leader_election.cc:304] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ac80df82774b9b87c0be076b5637af; no voters: ae949d1fcdf54e87930871ac0ae7da31, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:18.741557 4597 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:57:18.939514 4598 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:18.955035 4598 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:18.965708 4598 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:18.965982 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:18.966627 4409 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:18.968222 4464 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31; no voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:18.973999 4598 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:57:19.504644 4599 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:19.505306 4599 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:19.509974 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:19.513118 4409 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:19.514580 4464 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31; no voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:19.523002 4599 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:19.524219 4599 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 19:57:18.492698 4585 debug-util.cc:398] Leaking SignalData structure 0x7b08000186a0 after lost signal to thread 4562
W20250626 19:57:19.592348 4562 thread.cc:640] GCE (cloud detector) Time spent creating pthread: real 2.497s user 0.583s sys 1.117s
W20250626 19:57:19.597689 4562 thread.cc:606] GCE (cloud detector) Time spent starting thread: real 2.503s user 0.583s sys 1.117s
W20250626 19:57:17.099118 4587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:57:19.607035 4562 server_base.cc:1048] running on GCE node
W20250626 19:57:19.611752 4602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:57:19.613248 4562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 19:57:19.623230 4562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 19:57:19.626951 4562 hybrid_clock.cc:648] HybridClock initialized: now 1750967839626909 us; error 45 us; skew 500 ppm
I20250626 19:57:19.628113 4562 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 19:57:19.659512 4562 webserver.cc:469] Webserver started at http://127.4.46.129:35767/ using document root <none> and password file <none>
I20250626 19:57:19.660933 4562 fs_manager.cc:362] Metadata directory not provided
I20250626 19:57:19.661307 4562 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 19:57:19.661933 4562 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 19:57:19.668646 4562 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/data/instance:
uuid: "4d3be519a0044a2f921a503412f72e86"
format_stamp: "Formatted at 2025-06-26 19:57:19 on dist-test-slave-48dm"
I20250626 19:57:19.670244 4562 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/instance:
uuid: "4d3be519a0044a2f921a503412f72e86"
format_stamp: "Formatted at 2025-06-26 19:57:19 on dist-test-slave-48dm"
I20250626 19:57:19.696906 4562 fs_manager.cc:696] Time spent creating directory manager: real 0.026s user 0.011s sys 0.013s
I20250626 19:57:19.708220 4607 raft_consensus.cc:491] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:19.709009 4607 raft_consensus.cc:513] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:19.710315 4608 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 19:57:19.711189 4607 leader_election.cc:290] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:19.712692 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ac80df82774b9b87c0be076b5637af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:19.713758 4488 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ac80df82774b9b87c0be076b5637af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31" is_pre_election: true
I20250626 19:57:19.715716 4383 leader_election.cc:304] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ac80df82774b9b87c0be076b5637af; no voters: ae949d1fcdf54e87930871ac0ae7da31, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:19.717495 4562 fs_manager.cc:730] Time spent opening block manager: real 0.016s user 0.006s sys 0.004s
I20250626 19:57:19.718130 4562 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal
uuid: "4d3be519a0044a2f921a503412f72e86"
format_stamp: "Formatted at 2025-06-26 19:57:19 on dist-test-slave-48dm"
I20250626 19:57:19.718662 4562 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 19:57:19.722909 4607 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:57:19.790288 4562 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 19:57:19.792157 4562 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 19:57:19.792649 4562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 19:57:19.797475 4562 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 19:57:19.805435 4562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 19:57:19.805835 4562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250626 19:57:19.806160 4562 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 19:57:19.806373 4562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 19:57:20.080327 4562 rpc_server.cc:307] RPC server started. Bound to: 127.4.46.129:39845
I20250626 19:57:20.080456 4721 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.46.129:39845 every 8 connection(s)
I20250626 19:57:20.083842 4562 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/data/info.pb
I20250626 19:57:20.087740 4282 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu as pid 4562
I20250626 19:57:20.088223 4282 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/instance
I20250626 19:57:20.105592 4282 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
/tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.46.130:0
--local_ip_for_outbound_sockets=127.4.46.130
--webserver_interface=127.4.46.130
--webserver_port=0
--tserver_master_addrs=127.4.46.190:33823,127.4.46.189:45039,127.4.46.188:40801
--builtin_ntp_servers=127.4.46.148:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250626 19:57:20.135969 4722 heartbeater.cc:344] Connected to a master server at 127.4.46.188:40801
I20250626 19:57:20.136543 4722 heartbeater.cc:461] Registering TS with master...
I20250626 19:57:20.137990 4722 heartbeater.cc:507] Master 127.4.46.188:40801 requested a full tablet report, sending...
I20250626 19:57:20.142913 4478 ts_manager.cc:194] Registered new tserver with Master: 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:20.179937 4726 heartbeater.cc:344] Connected to a master server at 127.4.46.189:45039
I20250626 19:57:20.180311 4726 heartbeater.cc:461] Registering TS with master...
I20250626 19:57:20.181146 4726 heartbeater.cc:507] Master 127.4.46.189:45039 requested a full tablet report, sending...
I20250626 19:57:20.184540 4399 ts_manager.cc:194] Registered new tserver with Master: 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:20.187134 4727 heartbeater.cc:344] Connected to a master server at 127.4.46.190:33823
I20250626 19:57:20.187464 4727 heartbeater.cc:461] Registering TS with master...
I20250626 19:57:20.188338 4727 heartbeater.cc:507] Master 127.4.46.190:33823 requested a full tablet report, sending...
I20250626 19:57:20.191234 4323 ts_manager.cc:194] Registered new tserver with Master: 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:20.193706 4323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.46.129:42245
W20250626 19:57:20.595613 4725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 19:57:20.596249 4725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 19:57:20.597271 4725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 19:57:20.654655 4725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 19:57:20.656265 4725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.46.130
I20250626 19:57:20.700198 4727 heartbeater.cc:499] Master 127.4.46.190:33823 was elected leader, sending a full tablet report...
I20250626 19:57:20.704589 4725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.46.148:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.46.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.46.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.46.190:33823,127.4.46.189:45039,127.4.46.188:40801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.46.130
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6785
TSAN enabled
I20250626 19:57:20.705909 4725 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 19:57:20.707501 4725 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 19:57:20.725793 4741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:57:20.910087 4745 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:20.910863 4745 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:20.915153 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:20.918736 4409 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:20.920574 4464 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31; no voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:20.926132 4745 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:20.927109 4745 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:57:21.480312 4747 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:21.480971 4747 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:21.483507 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:21.487294 4409 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:21.488687 4464 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31; no voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:21.494235 4747 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:21.495113 4747 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 19:57:20.728915 4738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:57:20.726119 4739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:57:22.159360 4740 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1435 milliseconds
I20250626 19:57:22.159411 4725 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 19:57:22.164341 4725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 19:57:22.167706 4725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 19:57:22.169209 4725 hybrid_clock.cc:648] HybridClock initialized: now 1750967842169159 us; error 42 us; skew 500 ppm
I20250626 19:57:22.170498 4725 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 19:57:22.178560 4725 webserver.cc:469] Webserver started at http://127.4.46.130:42243/ using document root <none> and password file <none>
I20250626 19:57:22.179893 4725 fs_manager.cc:362] Metadata directory not provided
I20250626 19:57:22.180131 4725 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 19:57:22.180681 4725 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 19:57:22.186378 4725 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/data/instance:
uuid: "93f885c6802341c889a40e9c20fbfff1"
format_stamp: "Formatted at 2025-06-26 19:57:22 on dist-test-slave-48dm"
I20250626 19:57:22.187894 4725 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/instance:
uuid: "93f885c6802341c889a40e9c20fbfff1"
format_stamp: "Formatted at 2025-06-26 19:57:22 on dist-test-slave-48dm"
I20250626 19:57:22.197933 4725 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20250626 19:57:22.204952 4753 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 19:57:22.206333 4725 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250626 19:57:22.206723 4725 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal
uuid: "93f885c6802341c889a40e9c20fbfff1"
format_stamp: "Formatted at 2025-06-26 19:57:22 on dist-test-slave-48dm"
I20250626 19:57:22.207151 4725 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 19:57:22.258481 4725 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 19:57:22.260126 4725 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 19:57:22.260543 4725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 19:57:22.263640 4725 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 19:57:22.267812 4725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 19:57:22.268019 4725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 19:57:22.268219 4725 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 19:57:22.268350 4725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 19:57:22.386816 4836 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:22.387427 4836 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:22.390578 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:22.390857 4409 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:22.392412 4464 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31; no voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:22.392704 4836 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:22.393118 4836 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:57:22.433591 4725 rpc_server.cc:307] RPC server started. Bound to: 127.4.46.130:39951
I20250626 19:57:22.433691 4867 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.46.130:39951 every 8 connection(s)
I20250626 19:57:22.436126 4725 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/data/info.pb
I20250626 19:57:22.438578 4282 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu as pid 4725
I20250626 19:57:22.439363 4282 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/instance
I20250626 19:57:22.452188 4282 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
/tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.46.131:0
--local_ip_for_outbound_sockets=127.4.46.131
--webserver_interface=127.4.46.131
--webserver_port=0
--tserver_master_addrs=127.4.46.190:33823,127.4.46.189:45039,127.4.46.188:40801
--builtin_ntp_servers=127.4.46.148:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250626 19:57:22.490458 4869 heartbeater.cc:344] Connected to a master server at 127.4.46.189:45039
I20250626 19:57:22.490936 4869 heartbeater.cc:461] Registering TS with master...
I20250626 19:57:22.492117 4869 heartbeater.cc:507] Master 127.4.46.189:45039 requested a full tablet report, sending...
I20250626 19:57:22.492637 4870 heartbeater.cc:344] Connected to a master server at 127.4.46.190:33823
I20250626 19:57:22.492939 4870 heartbeater.cc:461] Registering TS with master...
I20250626 19:57:22.493803 4870 heartbeater.cc:507] Master 127.4.46.190:33823 requested a full tablet report, sending...
I20250626 19:57:22.495347 4399 ts_manager.cc:194] Registered new tserver with Master: 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:22.496135 4868 heartbeater.cc:344] Connected to a master server at 127.4.46.188:40801
I20250626 19:57:22.496464 4868 heartbeater.cc:461] Registering TS with master...
I20250626 19:57:22.496470 4323 ts_manager.cc:194] Registered new tserver with Master: 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:22.497371 4868 heartbeater.cc:507] Master 127.4.46.188:40801 requested a full tablet report, sending...
I20250626 19:57:22.498237 4323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.46.130:59377
I20250626 19:57:22.500430 4478 ts_manager.cc:194] Registered new tserver with Master: 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:22.779508 4875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 19:57:22.780035 4875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 19:57:22.780640 4875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 19:57:22.811563 4875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 19:57:22.812546 4875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.46.131
I20250626 19:57:22.848430 4875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.46.148:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.46.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.46.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.46.190:33823,127.4.46.189:45039,127.4.46.188:40801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.46.131
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6785
TSAN enabled
I20250626 19:57:22.849812 4875 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 19:57:22.851433 4875 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 19:57:22.867549 4889 raft_consensus.cc:491] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:22.868301 4889 raft_consensus.cc:513] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
W20250626 19:57:22.869447 4886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:57:22.871325 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ac80df82774b9b87c0be076b5637af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:22.872699 4889 leader_election.cc:290] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:22.872610 4488 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ac80df82774b9b87c0be076b5637af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31" is_pre_election: true
I20250626 19:57:22.874679 4383 leader_election.cc:304] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ac80df82774b9b87c0be076b5637af; no voters: ae949d1fcdf54e87930871ac0ae7da31, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:22.875686 4889 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:57:22.902559 4892 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:22.904045 4892 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:22.906535 4892 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:22.907425 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:22.907913 4409 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:22.908663 4409 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae949d1fcdf54e87930871ac0ae7da31 in term 1.
I20250626 19:57:22.909771 4464 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62ac80df82774b9b87c0be076b5637af, ae949d1fcdf54e87930871ac0ae7da31; no voters: e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:22.910429 4892 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 19:57:22.910769 4892 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:22.911047 4892 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:57:22.915925 4892 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:22.917418 4892 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 election: Requested vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:22.918237 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360"
I20250626 19:57:22.918912 4409 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af"
I20250626 19:57:22.919438 4409 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:57:22.926088 4409 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae949d1fcdf54e87930871ac0ae7da31 in term 2.
I20250626 19:57:22.926999 4464 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62ac80df82774b9b87c0be076b5637af, ae949d1fcdf54e87930871ac0ae7da31; no voters: e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:22.927627 4892 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 19:57:22.928681 4892 raft_consensus.cc:695] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 2 LEADER]: Becoming Leader. State: Replica: ae949d1fcdf54e87930871ac0ae7da31, State: Running, Role: LEADER
I20250626 19:57:22.929654 4892 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:22.934252 4893 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae949d1fcdf54e87930871ac0ae7da31. Latest consensus state: current_term: 2 leader_uuid: "ae949d1fcdf54e87930871ac0ae7da31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:22.934983 4893 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: This master's current role is: LEADER
I20250626 19:57:22.937258 4895 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 19:57:22.942363 4895 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 19:57:22.944837 4895 catalog_manager.cc:1261] Loaded cluster ID: 845ffa7fa66f40c8abadc28aed775425
I20250626 19:57:22.945040 4895 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 19:57:22.947919 4895 catalog_manager.cc:1506] Loading token signing keys...
I20250626 19:57:22.951748 4895 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31: Loaded TSK: 0
I20250626 19:57:22.953445 4895 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 19:57:22.961261 4409 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 2 FOLLOWER]: Rejecting Update request from peer e6c58006b6be41c1bd6bdb5eb010d360 for earlier term 1. Current term is 2. Ops: []
I20250626 19:57:22.962993 4896 consensus_queue.cc:1046] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 19:57:22.964807 4897 raft_consensus.cc:3053] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 19:57:22.965142 4897 raft_consensus.cc:738] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e6c58006b6be41c1bd6bdb5eb010d360, State: Running, Role: LEADER
I20250626 19:57:22.965859 4897 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:22.966856 4897 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:57:23.001832 4870 heartbeater.cc:499] Master 127.4.46.190:33823 was elected leader, sending a full tablet report...
I20250626 19:57:23.013670 4547 catalog_manager.cc:1604] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 19:57:23.463150 4897 raft_consensus.cc:491] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:23.463789 4897 raft_consensus.cc:513] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:23.466289 4897 leader_election.cc:290] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:23.467655 4409 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:23.469045 4409 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6c58006b6be41c1bd6bdb5eb010d360 in term 2.
I20250626 19:57:23.470062 4488 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31" is_pre_election: true
I20250626 19:57:23.471112 4310 leader_election.cc:304] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360; no voters:
I20250626 19:57:23.472599 4897 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 19:57:23.473129 4897 raft_consensus.cc:491] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:23.473615 4897 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 2 FOLLOWER]: Advancing to term 3
I20250626 19:57:23.484640 4897 raft_consensus.cc:513] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:23.489058 4409 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 2 FOLLOWER]: Refusing update from remote peer ae949d1fcdf54e87930871ac0ae7da31: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 19:57:23.490679 4488 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31"
I20250626 19:57:23.492048 4898 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 19:57:23.494252 4897 leader_election.cc:290] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 3 election: Requested vote from peers 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:23.496922 4407 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af"
I20250626 19:57:23.499258 4310 leader_election.cc:304] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6c58006b6be41c1bd6bdb5eb010d360; no voters: 62ac80df82774b9b87c0be076b5637af, ae949d1fcdf54e87930871ac0ae7da31
I20250626 19:57:23.505820 4868 heartbeater.cc:499] Master 127.4.46.188:40801 was elected leader, sending a full tablet report...
I20250626 19:57:23.500882 4899 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 19:57:23.517848 4334 raft_consensus.cc:1238] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 3 FOLLOWER]: Rejecting Update request from peer ae949d1fcdf54e87930871ac0ae7da31 for earlier term 2. Current term is 3. Ops: []
I20250626 19:57:23.526373 4898 consensus_queue.cc:1046] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 19:57:23.531468 4898 raft_consensus.cc:3053] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 19:57:23.532058 4898 raft_consensus.cc:738] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae949d1fcdf54e87930871ac0ae7da31, State: Running, Role: LEADER
I20250626 19:57:23.534987 4898 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:23.536857 4898 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 2 FOLLOWER]: Advancing to term 3
I20250626 19:57:23.552865 4900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae949d1fcdf54e87930871ac0ae7da31. Latest consensus state: current_term: 2 leader_uuid: "ae949d1fcdf54e87930871ac0ae7da31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:23.554037 4900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:23.669160 4722 heartbeater.cc:499] Master 127.4.46.188:40801 was elected leader, sending a full tablet report...
I20250626 19:57:23.994587 4898 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:23.995313 4898 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:23.998610 4898 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:23.998629 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:23.999265 4334 raft_consensus.cc:2466] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae949d1fcdf54e87930871ac0ae7da31 in term 3.
I20250626 19:57:24.000452 4462 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31, e6c58006b6be41c1bd6bdb5eb010d360; no voters:
I20250626 19:57:24.001288 4898 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 19:57:24.000766 4409 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:24.001725 4898 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:24.002219 4898 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 3 FOLLOWER]: Advancing to term 4
I20250626 19:57:24.014127 4898 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:24.017516 4898 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 4 election: Requested vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:24.018276 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360"
I20250626 19:57:24.019024 4334 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 3 FOLLOWER]: Advancing to term 4
I20250626 19:57:24.023766 4409 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af"
I20250626 19:57:24.028062 4334 raft_consensus.cc:2466] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae949d1fcdf54e87930871ac0ae7da31 in term 4.
I20250626 19:57:24.029313 4462 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31, e6c58006b6be41c1bd6bdb5eb010d360; no voters: 62ac80df82774b9b87c0be076b5637af
I20250626 19:57:24.031090 4898 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 19:57:24.032552 4898 raft_consensus.cc:695] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 4 LEADER]: Becoming Leader. State: Replica: ae949d1fcdf54e87930871ac0ae7da31, State: Running, Role: LEADER
I20250626 19:57:24.033558 4898 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:24.038028 4903 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae949d1fcdf54e87930871ac0ae7da31. Latest consensus state: current_term: 4 leader_uuid: "ae949d1fcdf54e87930871ac0ae7da31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.038849 4903 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: This master's current role is: LEADER
I20250626 19:57:24.040534 4905 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 19:57:24.050488 4900 raft_consensus.cc:491] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae949d1fcdf54e87930871ac0ae7da31)
I20250626 19:57:24.050629 4905 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 19:57:24.050990 4900 raft_consensus.cc:513] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:24.052831 4900 leader_election.cc:290] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:24.055006 4488 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ac80df82774b9b87c0be076b5637af" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31" is_pre_election: true
I20250626 19:57:24.055712 4905 catalog_manager.cc:1261] Loaded cluster ID: 845ffa7fa66f40c8abadc28aed775425
I20250626 19:57:24.056035 4905 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 19:57:24.056368 4334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ac80df82774b9b87c0be076b5637af" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:24.057070 4334 raft_consensus.cc:2366] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62ac80df82774b9b87c0be076b5637af for earlier term 3. Current term is 4.
I20250626 19:57:24.058799 4383 leader_election.cc:400] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 3 pre-election: Vote denied by peer e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62ac80df82774b9b87c0be076b5637af for earlier term 3. Current term is 4.
I20250626 19:57:24.059126 4383 leader_election.cc:403] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250626 19:57:24.060356 4900 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 2 FOLLOWER]: Advancing to term 4
I20250626 19:57:24.065367 4905 catalog_manager.cc:1506] Loading token signing keys...
I20250626 19:57:24.069535 4900 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62ac80df82774b9b87c0be076b5637af for earlier term 3. Current term is 4.
I20250626 19:57:24.071877 4905 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31: Loaded TSK: 0
I20250626 19:57:24.072983 4905 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250626 19:57:22.869735 4888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:57:22.870577 4885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:57:24.189723 4887 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1320 milliseconds
I20250626 19:57:24.189851 4875 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 19:57:24.191186 4875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 19:57:24.193308 4875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 19:57:24.194648 4875 hybrid_clock.cc:648] HybridClock initialized: now 1750967844194615 us; error 57 us; skew 500 ppm
I20250626 19:57:24.195492 4875 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 19:57:24.202149 4875 webserver.cc:469] Webserver started at http://127.4.46.131:43171/ using document root <none> and password file <none>
I20250626 19:57:24.203164 4875 fs_manager.cc:362] Metadata directory not provided
I20250626 19:57:24.203370 4875 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 19:57:24.203855 4875 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 19:57:24.208271 4875 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/data/instance:
uuid: "5f8fb6d535ea42359fe34478df9e00a6"
format_stamp: "Formatted at 2025-06-26 19:57:24 on dist-test-slave-48dm"
I20250626 19:57:24.209398 4875 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/instance:
uuid: "5f8fb6d535ea42359fe34478df9e00a6"
format_stamp: "Formatted at 2025-06-26 19:57:24 on dist-test-slave-48dm"
I20250626 19:57:24.216607 4875 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20250626 19:57:24.222791 4910 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 19:57:24.223984 4875 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250626 19:57:24.224306 4875 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal
uuid: "5f8fb6d535ea42359fe34478df9e00a6"
format_stamp: "Formatted at 2025-06-26 19:57:24 on dist-test-slave-48dm"
I20250626 19:57:24.224637 4875 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 19:57:24.278023 4875 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 19:57:24.279975 4875 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 19:57:24.280412 4875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 19:57:24.282843 4875 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 19:57:24.287312 4875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 19:57:24.287528 4875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 19:57:24.287757 4875 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 19:57:24.287921 4875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 19:57:24.441519 4875 rpc_server.cc:307] RPC server started. Bound to: 127.4.46.131:40711
I20250626 19:57:24.441603 5023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.46.131:40711 every 8 connection(s)
I20250626 19:57:24.445240 4875 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/data/info.pb
I20250626 19:57:24.446208 4282 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu as pid 4875
I20250626 19:57:24.446614 4282 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/instance
I20250626 19:57:24.496639 5025 heartbeater.cc:344] Connected to a master server at 127.4.46.189:45039
I20250626 19:57:24.497319 5025 heartbeater.cc:461] Registering TS with master...
I20250626 19:57:24.498883 5025 heartbeater.cc:507] Master 127.4.46.189:45039 requested a full tablet report, sending...
I20250626 19:57:24.499680 5024 heartbeater.cc:344] Connected to a master server at 127.4.46.188:40801
I20250626 19:57:24.501905 5024 heartbeater.cc:461] Registering TS with master...
I20250626 19:57:24.502822 5024 heartbeater.cc:507] Master 127.4.46.188:40801 requested a full tablet report, sending...
I20250626 19:57:24.504601 5026 heartbeater.cc:344] Connected to a master server at 127.4.46.190:33823
I20250626 19:57:24.504521 4399 ts_manager.cc:194] Registered new tserver with Master: 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:24.504968 5026 heartbeater.cc:461] Registering TS with master...
I20250626 19:57:24.505846 5026 heartbeater.cc:507] Master 127.4.46.190:33823 requested a full tablet report, sending...
I20250626 19:57:24.505798 4478 ts_manager.cc:194] Registered new tserver with Master: 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:24.508339 4323 ts_manager.cc:194] Registered new tserver with Master: 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:24.509196 4478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.46.131:42505
I20250626 19:57:24.527801 4282 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 19:57:24.569509 4334 raft_consensus.cc:1273] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 4 FOLLOWER]: Refusing update from remote peer ae949d1fcdf54e87930871ac0ae7da31: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 19:57:24.571825 5041 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 19:57:24.590118 5043 raft_consensus.cc:491] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae949d1fcdf54e87930871ac0ae7da31)
I20250626 19:57:24.590700 5043 raft_consensus.cc:513] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:24.591959 5042 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae949d1fcdf54e87930871ac0ae7da31. Latest consensus state: current_term: 4 leader_uuid: "ae949d1fcdf54e87930871ac0ae7da31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.592837 5042 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:24.595431 4409 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:24.596041 4409 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6c58006b6be41c1bd6bdb5eb010d360 in term 4.
I20250626 19:57:24.596966 4488 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31" is_pre_election: true
I20250626 19:57:24.599366 4310 leader_election.cc:304] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360; no voters:
I20250626 19:57:24.600128 5042 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 19:57:24.600483 5043 leader_election.cc:290] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:24.600785 5042 raft_consensus.cc:491] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae949d1fcdf54e87930871ac0ae7da31)
I20250626 19:57:24.604063 5042 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 4 FOLLOWER]: Advancing to term 5
I20250626 19:57:24.613269 5041 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "ae949d1fcdf54e87930871ac0ae7da31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.614290 5041 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: This master's current role is: LEADER
I20250626 19:57:24.617435 4478 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51646:
name: "default.Table_18e79f64c3b240509949df78e2993144"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:24.621358 4409 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 4 FOLLOWER]: Refusing update from remote peer ae949d1fcdf54e87930871ac0ae7da31: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250626 19:57:24.629148 4478 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18e79f64c3b240509949df78e2993144 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:24.642910 5042 raft_consensus.cc:513] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:24.646921 5046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.005s
I20250626 19:57:24.647511 4477 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51646:
name: "default.Table_634df811aff54d3d97c5fddee19294a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:24.650592 4477 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_634df811aff54d3d97c5fddee19294a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:24.657480 4334 raft_consensus.cc:1238] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 5 FOLLOWER]: Rejecting Update request from peer ae949d1fcdf54e87930871ac0ae7da31 for earlier term 4. Current term is 5. Ops: []
I20250626 19:57:24.660337 5046 consensus_queue.cc:1046] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 19:57:24.661607 5049 raft_consensus.cc:3053] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 19:57:24.662169 5049 raft_consensus.cc:738] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae949d1fcdf54e87930871ac0ae7da31, State: Running, Role: LEADER
I20250626 19:57:24.663674 4407 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af"
I20250626 19:57:24.664664 5042 leader_election.cc:290] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 5 election: Requested vote from peers 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:24.665160 4488 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31"
I20250626 19:57:24.664271 5049 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:24.671963 5049 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 4 FOLLOWER]: Advancing to term 5
W20250626 19:57:24.673298 4478 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica ae949d1fcdf54e87930871ac0ae7da31 is not leader of this config: current role FOLLOWER
I20250626 19:57:24.678805 4722 heartbeater.cc:499] Master 127.4.46.188:40801 was elected leader, sending a full tablet report...
W20250626 19:57:24.683982 5034 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.46.188:40801)
I20250626 19:57:24.697081 5059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "ae949d1fcdf54e87930871ac0ae7da31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.698120 5059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:24.698665 5059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "ae949d1fcdf54e87930871ac0ae7da31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.699362 5059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:24.699491 5052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae949d1fcdf54e87930871ac0ae7da31. Latest consensus state: current_term: 4 leader_uuid: "ae949d1fcdf54e87930871ac0ae7da31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.700201 5052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:24.706466 4488 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6c58006b6be41c1bd6bdb5eb010d360 in term 5.
W20250626 19:57:24.715376 5051 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node ae949d1fcdf54e87930871ac0ae7da31 not leader. Raft Consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.721464 4308 leader_election.cc:304] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31, e6c58006b6be41c1bd6bdb5eb010d360; no voters: 62ac80df82774b9b87c0be076b5637af
I20250626 19:57:24.723038 5042 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 5 FOLLOWER]: Leader election won for term 5
W20250626 19:57:24.723309 4477 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: Replica ae949d1fcdf54e87930871ac0ae7da31 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.723639 5042 raft_consensus.cc:695] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 5 LEADER]: Becoming Leader. State: Replica: e6c58006b6be41c1bd6bdb5eb010d360, State: Running, Role: LEADER
I20250626 19:57:24.724638 5042 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:24.729449 5043 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6c58006b6be41c1bd6bdb5eb010d360. Latest consensus state: current_term: 5 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.730405 5043 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: This master's current role is: LEADER
I20250626 19:57:24.733299 5067 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 19:57:24.743391 5067 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 19:57:24.747277 5067 catalog_manager.cc:1261] Loaded cluster ID: 845ffa7fa66f40c8abadc28aed775425
I20250626 19:57:24.747643 5067 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 19:57:24.752017 5067 catalog_manager.cc:1506] Loading token signing keys...
I20250626 19:57:24.754421 5067 catalog_manager.cc:5966] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Loaded TSK: 0
I20250626 19:57:24.755306 5067 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 19:57:24.782442 4324 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_9c8ec84f6fdc4607b396b8b823d9ca77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:24.783291 4322 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_e9fc768f77874d6eb98639c1afff3f1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:24.783720 4321 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_634df811aff54d3d97c5fddee19294a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:24.782490 4323 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_18e79f64c3b240509949df78e2993144"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:24.786949 4324 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c8ec84f6fdc4607b396b8b823d9ca77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:57:24.787653 4321 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_634df811aff54d3d97c5fddee19294a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:57:24.787099 4323 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18e79f64c3b240509949df78e2993144 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:57:24.788102 4322 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9fc768f77874d6eb98639c1afff3f1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:24.806341 4488 raft_consensus.cc:1273] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 5 FOLLOWER]: Refusing update from remote peer e6c58006b6be41c1bd6bdb5eb010d360: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20250626 19:57:24.806341 4409 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 4 FOLLOWER]: Advancing to term 5
I20250626 19:57:24.808020 5043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250626 19:57:24.814862 4409 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 5 FOLLOWER]: Refusing update from remote peer e6c58006b6be41c1bd6bdb5eb010d360: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20250626 19:57:24.816504 5043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250626 19:57:24.818284 5049 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6c58006b6be41c1bd6bdb5eb010d360. Latest consensus state: current_term: 5 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.819206 5049 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:24.846109 5074 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.846935 5074 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: This master's current role is: LEADER
I20250626 19:57:24.846535 5057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6c58006b6be41c1bd6bdb5eb010d360. Latest consensus state: current_term: 5 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.847867 5057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:24.855507 4323 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_18e79f64c3b240509949df78e2993144" } new_table_name: "default.Table_e05a4332b01e4192ad95d5c0c2896a99" modify_external_catalogs: true
I20250626 19:57:24.858697 4322 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_e9fc768f77874d6eb98639c1afff3f1b" } new_table_name: "default.Table_a9daaddb188e4e13b52c9ea25ea8b2e5" modify_external_catalogs: true
I20250626 19:57:24.873211 5073 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.877466 5073 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: This master's current role is: LEADER
I20250626 19:57:24.877971 5074 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.880465 5074 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: This master's current role is: LEADER
I20250626 19:57:24.883215 5074 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.883924 5074 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: This master's current role is: LEADER
I20250626 19:57:24.886979 5049 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.887701 5049 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:24.907403 5057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:24.908102 5057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: This master's current role is: FOLLOWER
W20250626 19:57:24.925268 4322 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df3da12971654d9abc2ee8840c4f1592 has no consensus state
W20250626 19:57:24.922499 4323 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0db46a3da1a0486a9ec2a840d19260c1 has no consensus state
I20250626 19:57:24.935498 4323 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_9c8ec84f6fdc4607b396b8b823d9ca77" } modify_external_catalogs: true
I20250626 19:57:24.936558 4321 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_634df811aff54d3d97c5fddee19294a1" } modify_external_catalogs: true
I20250626 19:57:24.936262 4323 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_9c8ec84f6fdc4607b396b8b823d9ca77" } modify_external_catalogs: true
I20250626 19:57:24.937126 4321 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_634df811aff54d3d97c5fddee19294a1" } modify_external_catalogs: true
W20250626 19:57:24.995275 4308 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 0db46a3da1a0486a9ec2a840d19260c1 (table default.Table_e05a4332b01e4192ad95d5c0c2896a99 [id=0518f25b238c44d5a0e3c7530fa0c51b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0db46a3da1a0486a9ec2a840d19260c1 consensus state has no leader
I20250626 19:57:24.998777 4323 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet c021279c81554ba587da041b5fa0e12e
I20250626 19:57:25.003445 4321 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet b430a07f5c6c4a888233ab2107f78a8d
I20250626 19:57:25.012432 5024 heartbeater.cc:499] Master 127.4.46.188:40801 was elected leader, sending a full tablet report...
I20250626 19:57:25.027516 4868 heartbeater.cc:499] Master 127.4.46.188:40801 was elected leader, sending a full tablet report...
I20250626 19:57:25.030941 4800 tablet_service.cc:1468] Processing CreateTablet for tablet df3da12971654d9abc2ee8840c4f1592 (DEFAULT_TABLE table=default.Table_a9daaddb188e4e13b52c9ea25ea8b2e5 [id=21df672a819243f7a3d16bd6d0a599ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.032447 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df3da12971654d9abc2ee8840c4f1592. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:25.034467 4656 tablet_service.cc:1468] Processing CreateTablet for tablet df3da12971654d9abc2ee8840c4f1592 (DEFAULT_TABLE table=default.Table_a9daaddb188e4e13b52c9ea25ea8b2e5 [id=21df672a819243f7a3d16bd6d0a599ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.034869 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 0db46a3da1a0486a9ec2a840d19260c1 (DEFAULT_TABLE table=default.Table_e05a4332b01e4192ad95d5c0c2896a99 [id=0518f25b238c44d5a0e3c7530fa0c51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.032655 4657 tablet_service.cc:1468] Processing CreateTablet for tablet 0db46a3da1a0486a9ec2a840d19260c1 (DEFAULT_TABLE table=default.Table_e05a4332b01e4192ad95d5c0c2896a99 [id=0518f25b238c44d5a0e3c7530fa0c51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.035984 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db46a3da1a0486a9ec2a840d19260c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:25.036092 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df3da12971654d9abc2ee8840c4f1592. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:25.036834 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db46a3da1a0486a9ec2a840d19260c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:25.039629 4323 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_8694e21f21614dc39a47d853c613c2cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:25.041334 4323 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8694e21f21614dc39a47d853c613c2cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:25.044365 4958 tablet_service.cc:1468] Processing CreateTablet for tablet df3da12971654d9abc2ee8840c4f1592 (DEFAULT_TABLE table=default.Table_a9daaddb188e4e13b52c9ea25ea8b2e5 [id=21df672a819243f7a3d16bd6d0a599ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.045852 4321 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_419df62863ee4d20afc637b732f74d16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:25.046322 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df3da12971654d9abc2ee8840c4f1592. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:25.047525 4321 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_419df62863ee4d20afc637b732f74d16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:25.041844 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 0db46a3da1a0486a9ec2a840d19260c1 (DEFAULT_TABLE table=default.Table_e05a4332b01e4192ad95d5c0c2896a99 [id=0518f25b238c44d5a0e3c7530fa0c51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.073043 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db46a3da1a0486a9ec2a840d19260c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:25.074208 5089 tablet_bootstrap.cc:492] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:25.087352 5089 tablet_bootstrap.cc:654] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:25.089639 5089 log.cc:826] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1: Log is configured to *not* fsync() on all Append() calls
I20250626 19:57:25.113667 5089 tablet_bootstrap.cc:492] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:25.114326 5089 ts_tablet_manager.cc:1397] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.008s
I20250626 19:57:25.127566 4321 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_09c4af804f824bc1ab909d92d7c5bf5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:25.130086 4321 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09c4af804f824bc1ab909d92d7c5bf5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:25.135943 4323 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_419df62863ee4d20afc637b732f74d16" } new_table_name: "default.Table_dfb0d69f87b14418bd4aabb75c931b2b" modify_external_catalogs: true
I20250626 19:57:25.149464 4324 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_a9daaddb188e4e13b52c9ea25ea8b2e5" } new_table_name: "default.Table_f0f87fc9e3234d42aab1e5102ce8d137" modify_external_catalogs: true
I20250626 19:57:25.149268 4322 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_6705434694db47e78b1292a0cd30217b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:25.155972 4322 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6705434694db47e78b1292a0cd30217b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:25.164727 5091 tablet_bootstrap.cc:492] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:25.183975 5091 tablet_bootstrap.cc:654] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:25.187539 5091 log.cc:826] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: Log is configured to *not* fsync() on all Append() calls
W20250626 19:57:25.193954 4324 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df3da12971654d9abc2ee8840c4f1592 consensus state has no leader
I20250626 19:57:25.198243 5089 raft_consensus.cc:357] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.199834 5089 raft_consensus.cc:383] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:25.200220 5089 raft_consensus.cc:738] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:25.201517 5089 consensus_queue.cc:260] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.207479 5091 tablet_bootstrap.cc:492] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:25.208179 5091 ts_tablet_manager.cc:1397] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.009s
I20250626 19:57:25.231243 4870 heartbeater.cc:499] Master 127.4.46.190:33823 was elected leader, sending a full tablet report...
I20250626 19:57:25.238536 5089 ts_tablet_manager.cc:1428] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.124s user 0.031s sys 0.006s
W20250626 19:57:25.241143 4323 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4eb7f65e09642e3960e8672da517e1d has no consensus state
I20250626 19:57:25.250368 5089 tablet_bootstrap.cc:492] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:25.260388 4323 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_e05a4332b01e4192ad95d5c0c2896a99" } modify_external_catalogs: true
I20250626 19:57:25.261461 4323 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_e05a4332b01e4192ad95d5c0c2896a99" } modify_external_catalogs: true
I20250626 19:57:25.264350 5089 tablet_bootstrap.cc:654] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:25.266767 4324 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_8694e21f21614dc39a47d853c613c2cd" } modify_external_catalogs: true
I20250626 19:57:25.267592 4324 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_8694e21f21614dc39a47d853c613c2cd" } modify_external_catalogs: true
I20250626 19:57:25.279515 5089 tablet_bootstrap.cc:492] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:25.280054 5089 ts_tablet_manager.cc:1397] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.000s
I20250626 19:57:25.283265 5089 raft_consensus.cc:357] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.284154 5089 raft_consensus.cc:383] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:25.284509 5089 raft_consensus.cc:738] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:25.295046 5089 consensus_queue.cc:260] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.297343 5089 ts_tablet_manager.cc:1428] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250626 19:57:25.311157 5092 tablet_bootstrap.cc:492] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:25.317185 5091 raft_consensus.cc:357] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.318084 5091 raft_consensus.cc:383] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:25.318351 5091 raft_consensus.cc:738] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:25.319550 5091 consensus_queue.cc:260] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.330559 4320 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_413987923a93485aa2356930bf39dd64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:25.332219 4320 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_413987923a93485aa2356930bf39dd64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:25.335587 4319 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_889f18064aab4774b17bfa478a0a7e33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:25.337186 4319 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_889f18064aab4774b17bfa478a0a7e33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:25.342336 4727 heartbeater.cc:499] Master 127.4.46.190:33823 was elected leader, sending a full tablet report...
I20250626 19:57:25.346297 5091 ts_tablet_manager.cc:1428] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.138s user 0.043s sys 0.003s
I20250626 19:57:25.364583 4322 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 0db46a3da1a0486a9ec2a840d19260c1 (table default.Table_e05a4332b01e4192ad95d5c0c2896a99 [id=0518f25b238c44d5a0e3c7530fa0c51b]). Expected version 1 got 0
I20250626 19:57:25.365561 4322 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet df3da12971654d9abc2ee8840c4f1592 (table default.Table_f0f87fc9e3234d42aab1e5102ce8d137 [id=21df672a819243f7a3d16bd6d0a599ac]). Expected version 2 got 0
I20250626 19:57:25.370854 5092 tablet_bootstrap.cc:654] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:25.374022 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 1944bb9417a741e4adde9422ef5b8222 (DEFAULT_TABLE table=default.Table_8694e21f21614dc39a47d853c613c2cd [id=dcdd5d68fb8148aabc08e737a2cbabef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.375102 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1944bb9417a741e4adde9422ef5b8222. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:25.376940 4318 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet df3da12971654d9abc2ee8840c4f1592 (table default.Table_f0f87fc9e3234d42aab1e5102ce8d137 [id=21df672a819243f7a3d16bd6d0a599ac]). Expected version 2 got 0
I20250626 19:57:25.378284 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 1944bb9417a741e4adde9422ef5b8222 (DEFAULT_TABLE table=default.Table_8694e21f21614dc39a47d853c613c2cd [id=dcdd5d68fb8148aabc08e737a2cbabef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.379354 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1944bb9417a741e4adde9422ef5b8222. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:25.381796 5091 tablet_bootstrap.cc:492] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:25.386509 4655 tablet_service.cc:1468] Processing CreateTablet for tablet a4eb7f65e09642e3960e8672da517e1d (DEFAULT_TABLE table=default.Table_dfb0d69f87b14418bd4aabb75c931b2b [id=e970a60e4962403c9f0559ed5dd46115]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.387709 4655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4eb7f65e09642e3960e8672da517e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:25.391502 4801 tablet_service.cc:1468] Processing CreateTablet for tablet a4eb7f65e09642e3960e8672da517e1d (DEFAULT_TABLE table=default.Table_dfb0d69f87b14418bd4aabb75c931b2b [id=e970a60e4962403c9f0559ed5dd46115]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.392556 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4eb7f65e09642e3960e8672da517e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:25.392908 4318 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet df3da12971654d9abc2ee8840c4f1592 (table default.Table_f0f87fc9e3234d42aab1e5102ce8d137 [id=21df672a819243f7a3d16bd6d0a599ac]). Expected version 2 got 0
I20250626 19:57:25.396311 5089 tablet_bootstrap.cc:492] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:25.403461 5089 tablet_bootstrap.cc:654] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:25.405586 5091 tablet_bootstrap.cc:654] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:25.387480 4958 tablet_service.cc:1468] Processing CreateTablet for tablet 1944bb9417a741e4adde9422ef5b8222 (DEFAULT_TABLE table=default.Table_8694e21f21614dc39a47d853c613c2cd [id=dcdd5d68fb8148aabc08e737a2cbabef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.415369 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1944bb9417a741e4adde9422ef5b8222. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:25.418421 5089 tablet_bootstrap.cc:492] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:25.418859 5089 ts_tablet_manager.cc:1397] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.013s
I20250626 19:57:25.412814 4959 tablet_service.cc:1468] Processing CreateTablet for tablet a4eb7f65e09642e3960e8672da517e1d (DEFAULT_TABLE table=default.Table_dfb0d69f87b14418bd4aabb75c931b2b [id=e970a60e4962403c9f0559ed5dd46115]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.419824 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4eb7f65e09642e3960e8672da517e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:25.421319 5089 raft_consensus.cc:357] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.421965 5089 raft_consensus.cc:383] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:25.422236 5089 raft_consensus.cc:738] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
W20250626 19:57:25.432262 4729 tablet.cc:2378] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 19:57:25.441216 5089 consensus_queue.cc:260] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.443485 5089 ts_tablet_manager.cc:1428] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20250626 19:57:25.473551 5092 log.cc:826] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: Log is configured to *not* fsync() on all Append() calls
W20250626 19:57:25.474318 4876 tablet.cc:2378] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 19:57:25.478056 5091 tablet_bootstrap.cc:492] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:25.478515 5091 ts_tablet_manager.cc:1397] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.097s user 0.007s sys 0.009s
I20250626 19:57:25.481583 5091 raft_consensus.cc:357] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.482378 5091 raft_consensus.cc:383] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:25.484522 4321 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_dfb0d69f87b14418bd4aabb75c931b2b" } new_table_name: "default.Table_14cb4703932a4df9864af43b426af3fb" modify_external_catalogs: true
I20250626 19:57:25.486827 5092 tablet_bootstrap.cc:492] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:25.487335 5092 ts_tablet_manager.cc:1397] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.177s user 0.012s sys 0.006s
I20250626 19:57:25.487675 4318 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_6705434694db47e78b1292a0cd30217b" } new_table_name: "default.Table_d8774697646e49e9a77cfb883bae0f98" modify_external_catalogs: true
I20250626 19:57:25.492038 4320 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_126ea31ac2c2483e84cedd411a0247b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:25.493685 4320 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_126ea31ac2c2483e84cedd411a0247b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:25.482707 5091 raft_consensus.cc:738] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:25.495536 5091 consensus_queue.cc:260] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.500861 5091 ts_tablet_manager.cc:1428] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20250626 19:57:25.501788 5091 tablet_bootstrap.cc:492] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:25.508826 5096 raft_consensus.cc:491] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:25.509361 5096 raft_consensus.cc:513] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.516615 5096 leader_election.cc:290] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:25.517108 5026 heartbeater.cc:499] Master 127.4.46.190:33823 was elected leader, sending a full tablet report...
I20250626 19:57:25.524329 5091 tablet_bootstrap.cc:654] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:25.535616 5091 tablet_bootstrap.cc:492] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:25.536126 5091 ts_tablet_manager.cc:1397] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.004s
I20250626 19:57:25.539184 5091 raft_consensus.cc:357] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.540107 5091 raft_consensus.cc:383] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:25.540450 5091 raft_consensus.cc:738] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:25.541352 5091 consensus_queue.cc:260] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.543953 5091 ts_tablet_manager.cc:1428] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20250626 19:57:25.550791 5091 tablet_bootstrap.cc:492] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:25.556851 5091 tablet_bootstrap.cc:654] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:25.563750 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df3da12971654d9abc2ee8840c4f1592" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:25.564569 4821 raft_consensus.cc:2466] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:25.568109 4609 leader_election.cc:304] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:25.569375 5096 raft_consensus.cc:2802] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:25.569736 5096 raft_consensus.cc:491] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:25.570075 5096 raft_consensus.cc:3058] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:25.572613 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df3da12971654d9abc2ee8840c4f1592" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:25.575104 4611 leader_election.cc:343] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:25.577543 5096 raft_consensus.cc:513] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.579262 5096 leader_election.cc:290] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:25.580497 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df3da12971654d9abc2ee8840c4f1592" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
W20250626 19:57:25.585332 4611 leader_election.cc:343] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:25.588565 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df3da12971654d9abc2ee8840c4f1592" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:25.589097 4821 raft_consensus.cc:3058] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:25.624819 5092 raft_consensus.cc:357] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.627020 5092 raft_consensus.cc:383] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:25.627487 5092 raft_consensus.cc:738] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:25.633880 5092 consensus_queue.cc:260] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.598284 4821 raft_consensus.cc:2466] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:25.641304 5089 tablet_bootstrap.cc:492] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:25.642674 4609 leader_election.cc:304] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:25.644982 4319 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_3f8afab3426b4088b717c05b65bb84e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:25.646684 4319 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f8afab3426b4088b717c05b65bb84e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:25.647089 5096 raft_consensus.cc:2802] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:25.651703 5089 tablet_bootstrap.cc:654] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:25.663509 5094 raft_consensus.cc:491] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:25.664129 5094 raft_consensus.cc:513] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.680810 5094 leader_election.cc:290] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:25.682001 5096 raft_consensus.cc:695] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:25.684983 5089 tablet_bootstrap.cc:492] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:25.686324 4315 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_c5ff61c31334492c9cc8dede96094236"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:25.687525 5096 consensus_queue.cc:237] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
W20250626 19:57:25.688122 4315 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5ff61c31334492c9cc8dede96094236 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:25.690245 5089 ts_tablet_manager.cc:1397] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.008s
I20250626 19:57:25.693687 5089 raft_consensus.cc:357] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.698953 5092 ts_tablet_manager.cc:1428] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.211s user 0.036s sys 0.013s
I20250626 19:57:25.701422 5089 raft_consensus.cc:383] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:25.701742 5089 raft_consensus.cc:738] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:25.702546 5089 consensus_queue.cc:260] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.709666 5089 ts_tablet_manager.cc:1428] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.019s user 0.003s sys 0.004s
I20250626 19:57:25.731963 5091 tablet_bootstrap.cc:492] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:25.732498 5091 ts_tablet_manager.cc:1397] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.182s user 0.053s sys 0.044s
I20250626 19:57:25.735574 5091 raft_consensus.cc:357] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.736369 5091 raft_consensus.cc:383] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:25.736514 4317 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet df3da12971654d9abc2ee8840c4f1592 (table default.Table_f0f87fc9e3234d42aab1e5102ce8d137 [id=21df672a819243f7a3d16bd6d0a599ac]). Expected version 2 got 0
I20250626 19:57:25.736708 5091 raft_consensus.cc:738] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:25.737466 5091 consensus_queue.cc:260] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.737884 4317 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet a4eb7f65e09642e3960e8672da517e1d (table default.Table_dfb0d69f87b14418bd4aabb75c931b2b [id=e970a60e4962403c9f0559ed5dd46115]). Expected version 1 got 0
I20250626 19:57:25.738101 5092 tablet_bootstrap.cc:492] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:25.738946 4324 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Sending DeleteTablet for 3 replicas of tablet 1944bb9417a741e4adde9422ef5b8222
I20250626 19:57:25.741197 5091 ts_tablet_manager.cc:1428] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250626 19:57:25.742080 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet 1944bb9417a741e4adde9422ef5b8222 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:51044
I20250626 19:57:25.744985 4655 tablet_service.cc:1515] Processing DeleteTablet for tablet 1944bb9417a741e4adde9422ef5b8222 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:56472
I20250626 19:57:25.745330 4323 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Sending DeleteTablet for 3 replicas of tablet 0db46a3da1a0486a9ec2a840d19260c1
I20250626 19:57:25.748780 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet 0db46a3da1a0486a9ec2a840d19260c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:56472
W20250626 19:57:25.750509 5031 tablet.cc:2378] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 19:57:25.767091 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 0db46a3da1a0486a9ec2a840d19260c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:51044
W20250626 19:57:25.795423 4318 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3cc96a3e9c174cbca0196c5a1b27455c has no consensus state
I20250626 19:57:25.801882 5094 raft_consensus.cc:491] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:25.802385 5094 raft_consensus.cc:513] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.804338 5094 leader_election.cc:290] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:25.807407 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 1944bb9417a741e4adde9422ef5b8222 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:36858
I20250626 19:57:25.811247 4800 tablet_service.cc:1515] Processing DeleteTablet for tablet 0db46a3da1a0486a9ec2a840d19260c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:36858
W20250626 19:57:25.825424 4310 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 1944bb9417a741e4adde9422ef5b8222 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1944bb9417a741e4adde9422ef5b8222 already in progress: creating tablet
I20250626 19:57:25.827219 5112 tablet_replica.cc:331] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:25.828190 5112 raft_consensus.cc:2241] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:25.828815 5112 raft_consensus.cc:2270] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250626 19:57:25.839497 4310 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 0db46a3da1a0486a9ec2a840d19260c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0db46a3da1a0486a9ec2a840d19260c1 already in progress: creating tablet
I20250626 19:57:25.844977 4319 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_ef500e9a9c0c4ef69cb8f1905015a3f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:25.846696 4319 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef500e9a9c0c4ef69cb8f1905015a3f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:25.848634 5092 tablet_bootstrap.cc:654] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:25.856545 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db46a3da1a0486a9ec2a840d19260c1" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:25.858768 4677 raft_consensus.cc:2466] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:25.857098 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4eb7f65e09642e3960e8672da517e1d" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:25.860445 4676 raft_consensus.cc:2466] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:25.862149 4755 leader_election.cc:304] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:25.863168 5094 raft_consensus.cc:2802] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:25.855146 5112 ts_tablet_manager.cc:1905] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:25.863492 5094 raft_consensus.cc:491] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:25.863806 5094 raft_consensus.cc:3058] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:25.871626 5094 raft_consensus.cc:513] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.863512 4755 leader_election.cc:304] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:25.876770 5094 leader_election.cc:290] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:25.883064 5094 raft_consensus.cc:2802] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:25.883472 5094 raft_consensus.cc:491] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:25.883381 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db46a3da1a0486a9ec2a840d19260c1" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:25.883829 5094 raft_consensus.cc:3058] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:25.884008 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4eb7f65e09642e3960e8672da517e1d" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:25.884903 4756 leader_election.cc:343] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:25.885303 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db46a3da1a0486a9ec2a840d19260c1" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:25.888376 4323 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet df3da12971654d9abc2ee8840c4f1592 (table default.Table_f0f87fc9e3234d42aab1e5102ce8d137 [id=21df672a819243f7a3d16bd6d0a599ac]). Expected version 2 got 0
I20250626 19:57:25.889065 4323 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet a4eb7f65e09642e3960e8672da517e1d (table default.Table_dfb0d69f87b14418bd4aabb75c931b2b [id=e970a60e4962403c9f0559ed5dd46115]). Expected version 1 got 0
W20250626 19:57:25.889418 4756 leader_election.cc:343] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:25.889983 4756 leader_election.cc:343] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:25.892151 4655 tablet_service.cc:1468] Processing CreateTablet for tablet 3cc96a3e9c174cbca0196c5a1b27455c (DEFAULT_TABLE table=default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.893265 4655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cc96a3e9c174cbca0196c5a1b27455c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:25.893049 4320 catalog_manager.cc:5582] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:25.897032 4320 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet df3da12971654d9abc2ee8840c4f1592 (table default.Table_f0f87fc9e3234d42aab1e5102ce8d137 [id=21df672a819243f7a3d16bd6d0a599ac]). Expected version 2 got 0
I20250626 19:57:25.897806 4320 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet a4eb7f65e09642e3960e8672da517e1d (table default.Table_dfb0d69f87b14418bd4aabb75c931b2b [id=e970a60e4962403c9f0559ed5dd46115]). Expected version 1 got 0
W20250626 19:57:25.898639 4321 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4eb7f65e09642e3960e8672da517e1d consensus state has no leader
I20250626 19:57:25.901784 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db46a3da1a0486a9ec2a840d19260c1" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:25.902316 4676 raft_consensus.cc:3058] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:25.902217 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 3cc96a3e9c174cbca0196c5a1b27455c (DEFAULT_TABLE table=default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.903287 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cc96a3e9c174cbca0196c5a1b27455c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:25.905187 5094 raft_consensus.cc:513] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.908106 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4eb7f65e09642e3960e8672da517e1d" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:25.908118 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 3cc96a3e9c174cbca0196c5a1b27455c (DEFAULT_TABLE table=default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:25.908620 4677 raft_consensus.cc:3058] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:25.908993 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 0db46a3da1a0486a9ec2a840d19260c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:51044
I20250626 19:57:25.909145 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cc96a3e9c174cbca0196c5a1b27455c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:25.911820 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 1944bb9417a741e4adde9422ef5b8222 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:51044
I20250626 19:57:25.913739 4676 raft_consensus.cc:2466] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:25.916276 4755 leader_election.cc:304] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:25.918048 4677 raft_consensus.cc:2466] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:25.919054 4755 leader_election.cc:304] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:25.921391 5117 raft_consensus.cc:2802] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:25.922973 5115 tablet_replica.cc:331] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:25.924144 5115 raft_consensus.cc:2241] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:25.924997 5115 raft_consensus.cc:2270] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:25.933921 5118 raft_consensus.cc:2802] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:25.937397 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4eb7f65e09642e3960e8672da517e1d" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
W20250626 19:57:25.938643 4756 leader_election.cc:343] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:25.936880 5094 leader_election.cc:290] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:25.941224 5115 ts_tablet_manager.cc:1905] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:25.942070 4310 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 0db46a3da1a0486a9ec2a840d19260c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0db46a3da1a0486a9ec2a840d19260c1 already in progress: creating tablet
W20250626 19:57:25.944770 4310 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 1944bb9417a741e4adde9422ef5b8222 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1944bb9417a741e4adde9422ef5b8222 already in progress: creating tablet
I20250626 19:57:25.948086 5089 tablet_bootstrap.cc:492] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:25.953505 5092 tablet_bootstrap.cc:492] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:25.954088 5092 ts_tablet_manager.cc:1397] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.253s user 0.007s sys 0.005s
I20250626 19:57:25.957737 5089 tablet_bootstrap.cc:654] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:25.957381 5092 raft_consensus.cc:357] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.958066 5092 raft_consensus.cc:383] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:25.961915 5117 raft_consensus.cc:695] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:25.962893 5117 consensus_queue.cc:237] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.964562 4317 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_09c4af804f824bc1ab909d92d7c5bf5d" } modify_external_catalogs: true
I20250626 19:57:25.965090 4317 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_09c4af804f824bc1ab909d92d7c5bf5d" } modify_external_catalogs: true
I20250626 19:57:25.965399 5118 raft_consensus.cc:695] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:25.966159 5118 consensus_queue.cc:237] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.958344 5092 raft_consensus.cc:738] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:25.975409 5092 consensus_queue.cc:260] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.975919 5089 tablet_bootstrap.cc:492] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:25.975625 4321 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_c5ff61c31334492c9cc8dede96094236" } modify_external_catalogs: true
I20250626 19:57:25.976318 5104 raft_consensus.cc:491] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:25.976469 5089 ts_tablet_manager.cc:1397] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.000s
I20250626 19:57:25.976423 4321 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_c5ff61c31334492c9cc8dede96094236" } modify_external_catalogs: true
I20250626 19:57:25.976823 5104 raft_consensus.cc:513] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:25.986145 5104 leader_election.cc:290] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:25.987380 5089 raft_consensus.cc:357] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:25.988241 5089 raft_consensus.cc:383] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:25.988574 5089 raft_consensus.cc:738] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:25.990327 4323 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet df3da12971654d9abc2ee8840c4f1592 (table default.Table_f0f87fc9e3234d42aab1e5102ce8d137 [id=21df672a819243f7a3d16bd6d0a599ac]). Expected version 2 got 0
I20250626 19:57:26.001880 4323 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_f0f87fc9e3234d42aab1e5102ce8d137" } new_table_name: "default.Table_20308edcc9354ee4893ae464106f8702" modify_external_catalogs: true
I20250626 19:57:26.004325 5091 tablet_bootstrap.cc:492] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:26.005966 4319 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_89c4b3541e7c4b08b25f28abf471d42d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:26.007014 4308 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 3cc96a3e9c174cbca0196c5a1b27455c (table default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3cc96a3e9c174cbca0196c5a1b27455c consensus state has no leader
I20250626 19:57:26.012253 5092 ts_tablet_manager.cc:1428] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.058s user 0.006s sys 0.001s
I20250626 19:57:26.013123 5092 tablet_bootstrap.cc:492] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
W20250626 19:57:26.013177 4319 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89c4b3541e7c4b08b25f28abf471d42d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:26.017256 5091 tablet_bootstrap.cc:654] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:26.020510 5089 consensus_queue.cc:260] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:26.022425 5117 raft_consensus.cc:491] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:26.023102 5089 ts_tablet_manager.cc:1428] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20250626 19:57:26.022851 5117 raft_consensus.cc:513] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:26.026351 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3cc96a3e9c174cbca0196c5a1b27455c" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:26.027890 4755 leader_election.cc:343] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:26.028795 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3cc96a3e9c174cbca0196c5a1b27455c" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:26.030638 4756 leader_election.cc:343] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:26.031129 4756 leader_election.cc:304] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:26.032020 5120 raft_consensus.cc:2747] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:26.032101 5117 leader_election.cc:290] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:26.044600 5112 ts_tablet_manager.cc:1918] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:26.045081 5112 log.cc:1199] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/1944bb9417a741e4adde9422ef5b8222
I20250626 19:57:26.046022 5112 ts_tablet_manager.cc:1939] T 1944bb9417a741e4adde9422ef5b8222 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:26.046996 4315 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_a706c8809b574fa3ab8b00a36dfda5c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:26.049201 4315 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a706c8809b574fa3ab8b00a36dfda5c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:26.056177 5091 tablet_bootstrap.cc:492] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:26.056632 5091 ts_tablet_manager.cc:1397] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.004s
I20250626 19:57:26.059592 5091 raft_consensus.cc:357] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:26.060314 5091 raft_consensus.cc:383] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:26.061152 5091 raft_consensus.cc:738] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:26.061878 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet 0db46a3da1a0486a9ec2a840d19260c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:51044
I20250626 19:57:26.061959 5091 consensus_queue.cc:260] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:26.062858 5111 tablet_replica.cc:331] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:26.063679 5111 raft_consensus.cc:2241] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:26.064205 5111 raft_consensus.cc:2270] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:26.066826 5091 ts_tablet_manager.cc:1428] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20250626 19:57:26.067284 4317 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ee4747b8dd7949108c2e220a5be96e3a
I20250626 19:57:26.078454 5111 ts_tablet_manager.cc:1905] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:26.084172 4319 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_654b4c156965473190b6f9ae638705af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:26.084427 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db46a3da1a0486a9ec2a840d19260c1" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:26.085196 4676 raft_consensus.cc:2391] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 93f885c6802341c889a40e9c20fbfff1 in this term.
W20250626 19:57:26.086637 4319 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_654b4c156965473190b6f9ae638705af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:26.090822 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db46a3da1a0486a9ec2a840d19260c1" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:26.091598 4309 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 1944bb9417a741e4adde9422ef5b8222 (table default.Table_8694e21f21614dc39a47d853c613c2cd [id=dcdd5d68fb8148aabc08e737a2cbabef]) successfully deleted
I20250626 19:57:26.092578 4911 leader_election.cc:304] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
W20250626 19:57:26.093359 4911 raft_consensus.cc:2684] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250626 19:57:26.093578 5092 tablet_bootstrap.cc:654] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:26.097019 5112 tablet_replica.cc:331] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:26.097733 5112 raft_consensus.cc:2241] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:26.098155 5112 raft_consensus.cc:2270] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:26.100214 5112 ts_tablet_manager.cc:1905] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:26.110373 4318 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_413987923a93485aa2356930bf39dd64" } new_table_name: "default.Table_d867f59f19a048b6a533acb0236b876b" modify_external_catalogs: true
I20250626 19:57:26.116039 4316 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_b6e3c1f4224b4dee8c044d2548276942"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:26.117655 4316 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6e3c1f4224b4dee8c044d2548276942 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:26.118659 5092 tablet_bootstrap.cc:492] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:26.119083 5092 ts_tablet_manager.cc:1397] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.106s user 0.012s sys 0.000s
I20250626 19:57:26.124573 5092 raft_consensus.cc:357] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:26.125278 5092 raft_consensus.cc:383] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:26.125562 5092 raft_consensus.cc:738] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:26.131731 5092 consensus_queue.cc:260] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:26.146584 5115 ts_tablet_manager.cc:1918] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:26.149168 5111 ts_tablet_manager.cc:1918] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:26.154963 5111 log.cc:1199] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/0db46a3da1a0486a9ec2a840d19260c1
I20250626 19:57:26.153508 5092 ts_tablet_manager.cc:1428] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20250626 19:57:26.156466 5111 ts_tablet_manager.cc:1939] T 0db46a3da1a0486a9ec2a840d19260c1 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:26.158979 5092 tablet_bootstrap.cc:492] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:26.162628 5115 log.cc:1199] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/1944bb9417a741e4adde9422ef5b8222
I20250626 19:57:26.163817 5115 ts_tablet_manager.cc:1939] T 1944bb9417a741e4adde9422ef5b8222 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:26.166620 4977 raft_consensus.cc:3058] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:26.168099 4310 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 0db46a3da1a0486a9ec2a840d19260c1 (table default.Table_e05a4332b01e4192ad95d5c0c2896a99 [id=0518f25b238c44d5a0e3c7530fa0c51b]) successfully deleted
I20250626 19:57:26.170185 4821 raft_consensus.cc:1273] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Refusing update from remote peer 4d3be519a0044a2f921a503412f72e86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:57:26.174125 4977 raft_consensus.cc:1273] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Refusing update from remote peer 4d3be519a0044a2f921a503412f72e86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:57:26.173054 5096 consensus_queue.cc:1035] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250626 19:57:26.177044 4308 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 1944bb9417a741e4adde9422ef5b8222 (table default.Table_8694e21f21614dc39a47d853c613c2cd [id=dcdd5d68fb8148aabc08e737a2cbabef]) successfully deleted
I20250626 19:57:26.177587 5115 tablet_replica.cc:331] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:26.177532 5105 consensus_queue.cc:1035] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:26.179086 5115 raft_consensus.cc:2241] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:26.181831 5115 pending_rounds.cc:70] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1: Trying to abort 1 pending ops.
I20250626 19:57:26.182973 5115 pending_rounds.cc:77] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964297087922176 op_type: NO_OP noop_request { }
I20250626 19:57:26.184665 5115 raft_consensus.cc:2887] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:57:26.186417 5115 raft_consensus.cc:2270] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:26.165482 5112 ts_tablet_manager.cc:1918] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:26.193926 5112 log.cc:1199] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/0db46a3da1a0486a9ec2a840d19260c1
I20250626 19:57:26.196117 5115 ts_tablet_manager.cc:1905] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:26.194582 5112 ts_tablet_manager.cc:1939] T 0db46a3da1a0486a9ec2a840d19260c1 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:26.207650 5092 tablet_bootstrap.cc:654] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:26.214169 4317 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 3cc96a3e9c174cbca0196c5a1b27455c (table default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]). Expected version 1 got 0
I20250626 19:57:26.214540 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 5785508e37624f02bc2f6cc93007e3f8 (DEFAULT_TABLE table=default.Table_ef500e9a9c0c4ef69cb8f1905015a3f1 [id=95832380520e4cf18d762654fb684db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.215624 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5785508e37624f02bc2f6cc93007e3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.219070 5130 mvcc.cc:204] Tried to move back new op lower bound from 7171964297873895424 to 7171964295972286464. Current Snapshot: MvccSnapshot[applied={T|T < 7171964297873895424}]
I20250626 19:57:26.219488 4320 catalog_manager.cc:5582] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:26.220796 4320 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet a4eb7f65e09642e3960e8672da517e1d (table default.Table_14cb4703932a4df9864af43b426af3fb [id=e970a60e4962403c9f0559ed5dd46115]). Expected version 2 got 0
I20250626 19:57:26.223019 4309 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 0db46a3da1a0486a9ec2a840d19260c1 (table default.Table_e05a4332b01e4192ad95d5c0c2896a99 [id=0518f25b238c44d5a0e3c7530fa0c51b]) successfully deleted
I20250626 19:57:26.224448 4655 tablet_service.cc:1468] Processing CreateTablet for tablet 5785508e37624f02bc2f6cc93007e3f8 (DEFAULT_TABLE table=default.Table_ef500e9a9c0c4ef69cb8f1905015a3f1 [id=95832380520e4cf18d762654fb684db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.225853 4655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5785508e37624f02bc2f6cc93007e3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.228345 4657 tablet_service.cc:1468] Processing CreateTablet for tablet 612f87796cfa44ddbb19b02797031db9 (DEFAULT_TABLE table=default.Table_c5ff61c31334492c9cc8dede96094236 [id=23c193073c364d7c921797f0940caf10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.229000 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 5785508e37624f02bc2f6cc93007e3f8 (DEFAULT_TABLE table=default.Table_ef500e9a9c0c4ef69cb8f1905015a3f1 [id=95832380520e4cf18d762654fb684db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.229621 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 612f87796cfa44ddbb19b02797031db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.229907 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5785508e37624f02bc2f6cc93007e3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.229836 4958 tablet_service.cc:1468] Processing CreateTablet for tablet 612f87796cfa44ddbb19b02797031db9 (DEFAULT_TABLE table=default.Table_c5ff61c31334492c9cc8dede96094236 [id=23c193073c364d7c921797f0940caf10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.230885 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 612f87796cfa44ddbb19b02797031db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.239521 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 612f87796cfa44ddbb19b02797031db9 (DEFAULT_TABLE table=default.Table_c5ff61c31334492c9cc8dede96094236 [id=23c193073c364d7c921797f0940caf10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.239466 4957 tablet_service.cc:1468] Processing CreateTablet for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 (DEFAULT_TABLE table=default.Table_889f18064aab4774b17bfa478a0a7e33 [id=0df063bc889246c3ba841b1b2f8e6104]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.240614 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 612f87796cfa44ddbb19b02797031db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.240703 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.246899 5091 tablet_bootstrap.cc:492] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:26.267086 4799 tablet_service.cc:1468] Processing CreateTablet for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 (DEFAULT_TABLE table=default.Table_889f18064aab4774b17bfa478a0a7e33 [id=0df063bc889246c3ba841b1b2f8e6104]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.268779 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.269973 5091 tablet_bootstrap.cc:654] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:26.291409 4956 tablet_service.cc:1468] Processing CreateTablet for tablet fa93fa6c652f4aca848e1e4910fb8175 (DEFAULT_TABLE table=default.Table_126ea31ac2c2483e84cedd411a0247b4 [id=ae81e1731bc1461385bf84e443ff0730]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.292640 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa93fa6c652f4aca848e1e4910fb8175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.292915 5136 tablet.cc:1722] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250626 19:57:26.298962 5105 raft_consensus.cc:491] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:26.299449 5105 raft_consensus.cc:513] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:26.302266 5105 leader_election.cc:290] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:26.313211 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3cc96a3e9c174cbca0196c5a1b27455c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:26.315553 5135 tablet.cc:1722] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250626 19:57:26.317369 4821 raft_consensus.cc:2466] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:26.317540 4798 tablet_service.cc:1468] Processing CreateTablet for tablet fa93fa6c652f4aca848e1e4910fb8175 (DEFAULT_TABLE table=default.Table_126ea31ac2c2483e84cedd411a0247b4 [id=ae81e1731bc1461385bf84e443ff0730]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.325909 4798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa93fa6c652f4aca848e1e4910fb8175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.332845 4317 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_3f8afab3426b4088b717c05b65bb84e5" } modify_external_catalogs: true
I20250626 19:57:26.333416 4317 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_3f8afab3426b4088b717c05b65bb84e5" } modify_external_catalogs: true
I20250626 19:57:26.335445 4315 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_ad656f771f554bc98b2dc4db559e3acb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:26.335600 4609 leader_election.cc:304] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:26.336635 5096 raft_consensus.cc:2802] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250626 19:57:26.337059 4315 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad656f771f554bc98b2dc4db559e3acb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:26.337314 5096 raft_consensus.cc:491] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:26.337711 5096 raft_consensus.cc:3058] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:26.341045 4319 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 3cc96a3e9c174cbca0196c5a1b27455c (table default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]). Expected version 1 got 0
W20250626 19:57:26.344491 4318 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bd8dae5699c4725915814044d76a257 consensus state has no leader
I20250626 19:57:26.351547 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3cc96a3e9c174cbca0196c5a1b27455c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:26.355551 4656 tablet_service.cc:1468] Processing CreateTablet for tablet fa93fa6c652f4aca848e1e4910fb8175 (DEFAULT_TABLE table=default.Table_126ea31ac2c2483e84cedd411a0247b4 [id=ae81e1731bc1461385bf84e443ff0730]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.357103 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa93fa6c652f4aca848e1e4910fb8175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.371428 4319 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_a706c8809b574fa3ab8b00a36dfda5c4" } new_table_name: "default.Table_cc421ddfd27f48b2a8a3bf433f8720f8" modify_external_catalogs: true
W20250626 19:57:26.371807 4611 leader_election.cc:343] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:26.297183 4657 tablet_service.cc:1468] Processing CreateTablet for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 (DEFAULT_TABLE table=default.Table_889f18064aab4774b17bfa478a0a7e33 [id=0df063bc889246c3ba841b1b2f8e6104]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.380894 5089 tablet_bootstrap.cc:492] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:26.387347 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 3bd8dae5699c4725915814044d76a257 (DEFAULT_TABLE table=default.Table_413987923a93485aa2356930bf39dd64 [id=46d6d30275b2427a8591d9d3d6101194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.387979 4954 tablet_service.cc:1468] Processing CreateTablet for tablet ea7374ad6ba64e46b7df9852fa33f145 (DEFAULT_TABLE table=default.Table_3f8afab3426b4088b717c05b65bb84e5 [id=ca71ef30f2ff4908b3b558d31e1cf989]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.388677 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd8dae5699c4725915814044d76a257. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.389931 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7374ad6ba64e46b7df9852fa33f145. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.396088 4654 tablet_service.cc:1468] Processing CreateTablet for tablet 3bd8dae5699c4725915814044d76a257 (DEFAULT_TABLE table=default.Table_d867f59f19a048b6a533acb0236b876b [id=46d6d30275b2427a8591d9d3d6101194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.397981 5089 tablet_bootstrap.cc:654] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:26.374462 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.403330 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd8dae5699c4725915814044d76a257. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.403985 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 3bd8dae5699c4725915814044d76a257 (DEFAULT_TABLE table=default.Table_d867f59f19a048b6a533acb0236b876b [id=46d6d30275b2427a8591d9d3d6101194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.409170 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd8dae5699c4725915814044d76a257. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.417886 5092 tablet_bootstrap.cc:492] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:26.418469 5092 ts_tablet_manager.cc:1397] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.260s user 0.054s sys 0.115s
I20250626 19:57:26.421473 5092 raft_consensus.cc:357] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:26.422236 5092 raft_consensus.cc:383] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:26.422557 5092 raft_consensus.cc:738] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:26.423339 5092 consensus_queue.cc:260] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:26.425034 5115 ts_tablet_manager.cc:1918] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:26.426015 5092 ts_tablet_manager.cc:1428] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20250626 19:57:26.429111 5092 tablet_bootstrap.cc:492] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:26.434613 4318 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_2ab026a162784703bf424759f2114968"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:26.436316 4318 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ab026a162784703bf424759f2114968 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:26.438974 5115 log.cc:1199] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/0db46a3da1a0486a9ec2a840d19260c1
I20250626 19:57:26.443280 5115 ts_tablet_manager.cc:1939] T 0db46a3da1a0486a9ec2a840d19260c1 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:26.443517 4323 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_b2fa45f3c65e4e56b7617277fe9f16ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:26.445130 4323 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2fa45f3c65e4e56b7617277fe9f16ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:26.449188 4321 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Sending DeleteTablet for 3 replicas of tablet 612f87796cfa44ddbb19b02797031db9
I20250626 19:57:26.452628 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet 612f87796cfa44ddbb19b02797031db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:51044
I20250626 19:57:26.453464 4798 tablet_service.cc:1515] Processing DeleteTablet for tablet 612f87796cfa44ddbb19b02797031db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:36858
W20250626 19:57:26.454038 4310 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 612f87796cfa44ddbb19b02797031db9 because the tablet was not found. No further retry: Not found: Tablet not found: 612f87796cfa44ddbb19b02797031db9
I20250626 19:57:26.465878 5104 raft_consensus.cc:491] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:26.466322 5104 raft_consensus.cc:513] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:26.468058 5104 leader_election.cc:290] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:26.470574 5092 tablet_bootstrap.cc:654] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:26.479830 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4eb7f65e09642e3960e8672da517e1d" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:26.480535 4676 raft_consensus.cc:2391] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 93f885c6802341c889a40e9c20fbfff1 in this term.
I20250626 19:57:26.483193 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4eb7f65e09642e3960e8672da517e1d" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:26.485615 4798 tablet_service.cc:1468] Processing CreateTablet for tablet ea7374ad6ba64e46b7df9852fa33f145 (DEFAULT_TABLE table=default.Table_3f8afab3426b4088b717c05b65bb84e5 [id=ca71ef30f2ff4908b3b558d31e1cf989]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.493106 4308 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 0db46a3da1a0486a9ec2a840d19260c1 (table default.Table_e05a4332b01e4192ad95d5c0c2896a99 [id=0518f25b238c44d5a0e3c7530fa0c51b]) successfully deleted
W20250626 19:57:26.493774 4308 catalog_manager.cc:4908] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 612f87796cfa44ddbb19b02797031db9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 612f87796cfa44ddbb19b02797031db9 already in progress: creating tablet
I20250626 19:57:26.494264 4911 leader_election.cc:304] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:26.501235 4316 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: default.Table_20308edcc9354ee4893ae464106f8702 [id=21df672a819243f7a3d16bd6d0a599ac] alter complete (version 3)
I20250626 19:57:26.502853 4317 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Sending DeleteTablet for 3 replicas of tablet ea7374ad6ba64e46b7df9852fa33f145
I20250626 19:57:26.505553 5089 tablet_bootstrap.cc:492] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:26.506014 5089 ts_tablet_manager.cc:1397] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.125s user 0.005s sys 0.010s
I20250626 19:57:26.509037 4322 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet a4eb7f65e09642e3960e8672da517e1d (table default.Table_14cb4703932a4df9864af43b426af3fb [id=e970a60e4962403c9f0559ed5dd46115]). Expected version 2 got 0
I20250626 19:57:26.509048 5089 raft_consensus.cc:357] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:26.509730 5089 raft_consensus.cc:383] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:26.510043 5089 raft_consensus.cc:738] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:26.486655 4798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7374ad6ba64e46b7df9852fa33f145. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.513166 5091 tablet_bootstrap.cc:492] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:26.513581 5091 ts_tablet_manager.cc:1397] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.267s user 0.013s sys 0.004s
I20250626 19:57:26.513005 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 0db46a3da1a0486a9ec2a840d19260c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:36858
I20250626 19:57:26.514688 5104 raft_consensus.cc:3058] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:26.514988 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet ea7374ad6ba64e46b7df9852fa33f145 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:51044
W20250626 19:57:26.516494 4310 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet ea7374ad6ba64e46b7df9852fa33f145 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea7374ad6ba64e46b7df9852fa33f145 already in progress: creating tablet
W20250626 19:57:26.517988 4308 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 0db46a3da1a0486a9ec2a840d19260c1 because the tablet was not found. No further retry: Not found: Tablet not found: 0db46a3da1a0486a9ec2a840d19260c1
I20250626 19:57:26.523953 5091 raft_consensus.cc:357] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:26.524576 5091 raft_consensus.cc:383] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:26.524822 5091 raft_consensus.cc:738] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:26.525437 5091 consensus_queue.cc:260] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:26.525961 4799 tablet_service.cc:1515] Processing DeleteTablet for tablet ea7374ad6ba64e46b7df9852fa33f145 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:36858
I20250626 19:57:26.538825 5089 consensus_queue.cc:260] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:26.541846 5089 ts_tablet_manager.cc:1428] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.036s user 0.003s sys 0.004s
I20250626 19:57:26.543932 5092 tablet_bootstrap.cc:492] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:26.544446 5092 ts_tablet_manager.cc:1397] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.117s user 0.012s sys 0.000s
I20250626 19:57:26.544449 5096 raft_consensus.cc:513] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:26.547567 5092 raft_consensus.cc:357] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:26.548309 5092 raft_consensus.cc:383] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:26.548627 5092 raft_consensus.cc:738] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:26.549398 5092 consensus_queue.cc:260] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:26.553987 5096 leader_election.cc:290] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:26.563887 5089 tablet_bootstrap.cc:492] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
W20250626 19:57:26.564095 4308 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet ea7374ad6ba64e46b7df9852fa33f145 because the tablet was not found. No further retry: Not found: Tablet not found: ea7374ad6ba64e46b7df9852fa33f145
I20250626 19:57:26.551074 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3cc96a3e9c174cbca0196c5a1b27455c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:26.571044 4821 raft_consensus.cc:3058] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:26.573035 4611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 46707.6us
I20250626 19:57:26.575743 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3cc96a3e9c174cbca0196c5a1b27455c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:26.576320 4977 raft_consensus.cc:3058] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:26.578713 4821 raft_consensus.cc:2466] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:26.579883 4609 leader_election.cc:304] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:26.580991 5105 raft_consensus.cc:2802] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:26.581434 5105 raft_consensus.cc:695] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:26.582222 5105 consensus_queue.cc:237] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:26.582736 4977 raft_consensus.cc:2466] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:26.582713 5129 mvcc.cc:204] Tried to move back new op lower bound from 7171964297873895424 to 7171964295972286464. Current Snapshot: MvccSnapshot[applied={T|T < 7171964297873895424}]
I20250626 19:57:26.584599 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet ea7374ad6ba64e46b7df9852fa33f145 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:56472
I20250626 19:57:26.585839 4654 tablet_service.cc:1468] Processing CreateTablet for tablet ea7374ad6ba64e46b7df9852fa33f145 (DEFAULT_TABLE table=default.Table_3f8afab3426b4088b717c05b65bb84e5 [id=ca71ef30f2ff4908b3b558d31e1cf989]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.587383 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7374ad6ba64e46b7df9852fa33f145. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.597107 4676 raft_consensus.cc:1273] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Refusing update from remote peer 93f885c6802341c889a40e9c20fbfff1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 19:57:26.600106 4309 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet ea7374ad6ba64e46b7df9852fa33f145 because the tablet was not found. No further retry: Not found: Tablet not found: ea7374ad6ba64e46b7df9852fa33f145
I20250626 19:57:26.600351 5118 consensus_queue.cc:1035] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:26.601121 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 612f87796cfa44ddbb19b02797031db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:56472
W20250626 19:57:26.611356 4309 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 612f87796cfa44ddbb19b02797031db9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 612f87796cfa44ddbb19b02797031db9 already in progress: creating tablet
I20250626 19:57:26.628520 5092 ts_tablet_manager.cc:1428] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.084s user 0.006s sys 0.000s
I20250626 19:57:26.629647 5092 tablet_bootstrap.cc:492] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:26.632737 5135 tablet.cc:1717] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: Skipping requested alter to schema version 3, tablet already version 3
I20250626 19:57:26.635185 5136 tablet.cc:1717] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1: Skipping requested alter to schema version 3, tablet already version 3
I20250626 19:57:26.635566 5089 tablet_bootstrap.cc:654] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:26.653445 5091 ts_tablet_manager.cc:1428] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.140s user 0.002s sys 0.005s
I20250626 19:57:26.654520 5091 tablet_bootstrap.cc:492] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
W20250626 19:57:26.659608 4319 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 78a8f494f8614e7ebb8ca880b6c80e78 has no consensus state
I20250626 19:57:26.661523 5092 tablet_bootstrap.cc:654] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:26.662899 5104 raft_consensus.cc:2747] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:26.664186 4978 raft_consensus.cc:1273] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Refusing update from remote peer 93f885c6802341c889a40e9c20fbfff1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:57:26.668928 5146 consensus_queue.cc:1035] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:26.679529 5089 tablet_bootstrap.cc:492] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:26.680006 5089 ts_tablet_manager.cc:1397] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.116s user 0.011s sys 0.001s
I20250626 19:57:26.697113 5136 tablet.cc:1722] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250626 19:57:26.703586 5148 raft_consensus.cc:491] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:26.704281 5148 raft_consensus.cc:513] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:26.711982 5151 tablet.cc:1722] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250626 19:57:26.714259 4319 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_14cb4703932a4df9864af43b426af3fb" } new_table_name: "default.Table_f24d9839a6b84ea691c394b6b3ddf3b3" modify_external_catalogs: true
I20250626 19:57:26.723515 4315 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_603f3fe8b7224f19b395762b57f2fa5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:26.729460 4317 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_89c4b3541e7c4b08b25f28abf471d42d" } modify_external_catalogs: true
I20250626 19:57:26.730032 4317 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_89c4b3541e7c4b08b25f28abf471d42d" } modify_external_catalogs: true
I20250626 19:57:26.734160 4316 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_d92eac6ec7f04298acf37bef4ed302e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:26.743360 4322 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_b6e3c1f4224b4dee8c044d2548276942" } modify_external_catalogs: true
I20250626 19:57:26.743986 4322 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_b6e3c1f4224b4dee8c044d2548276942" } modify_external_catalogs: true
I20250626 19:57:26.745927 5089 raft_consensus.cc:357] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:26.746553 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet 612f87796cfa44ddbb19b02797031db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:56472
I20250626 19:57:26.747648 5112 tablet_replica.cc:331] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:26.748416 5112 raft_consensus.cc:2241] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:26.748886 5112 raft_consensus.cc:2270] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:26.746615 5089 raft_consensus.cc:383] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:26.751019 5089 raft_consensus.cc:738] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:26.751055 5112 ts_tablet_manager.cc:1905] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:26.751749 5089 consensus_queue.cc:260] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:26.753990 5089 ts_tablet_manager.cc:1428] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.074s user 0.006s sys 0.000s
I20250626 19:57:26.754866 5089 tablet_bootstrap.cc:492] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:26.759320 5154 tablet.cc:1722] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250626 19:57:26.759509 4755 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 63821.3us
W20250626 19:57:26.768639 4912 leader_election.cc:343] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Not found: Tablet not found: 1944bb9417a741e4adde9422ef5b8222
I20250626 19:57:26.787694 5112 ts_tablet_manager.cc:1918] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:26.788122 5112 log.cc:1199] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/612f87796cfa44ddbb19b02797031db9
I20250626 19:57:26.788808 5112 ts_tablet_manager.cc:1939] T 612f87796cfa44ddbb19b02797031db9 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
W20250626 19:57:26.775179 4911 leader_election.cc:343] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Not found: Tablet not found: 1944bb9417a741e4adde9422ef5b8222
I20250626 19:57:26.796485 5091 tablet_bootstrap.cc:654] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:26.796546 4911 leader_election.cc:304] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:26.797497 5104 raft_consensus.cc:2747] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:26.803740 4315 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_603f3fe8b7224f19b395762b57f2fa5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:26.803946 4309 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 612f87796cfa44ddbb19b02797031db9 (table default.Table_c5ff61c31334492c9cc8dede96094236 [id=23c193073c364d7c921797f0940caf10]) successfully deleted
W20250626 19:57:26.810669 4316 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d92eac6ec7f04298acf37bef4ed302e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:26.709422 5148 leader_election.cc:290] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:26.857458 5091 tablet_bootstrap.cc:492] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:26.857878 5091 ts_tablet_manager.cc:1397] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.204s user 0.011s sys 0.000s
I20250626 19:57:26.859560 5089 tablet_bootstrap.cc:654] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:26.874653 5135 tablet.cc:1722] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250626 19:57:26.882833 4320 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 3cc96a3e9c174cbca0196c5a1b27455c (table default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]). Expected version 1 got 0
I20250626 19:57:26.888813 5092 tablet_bootstrap.cc:492] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:26.889330 5092 ts_tablet_manager.cc:1397] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.260s user 0.007s sys 0.013s
I20250626 19:57:26.896189 4318 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet a4eb7f65e09642e3960e8672da517e1d (table default.Table_14cb4703932a4df9864af43b426af3fb [id=e970a60e4962403c9f0559ed5dd46115]). Expected version 2 got 0
I20250626 19:57:26.896898 4318 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet df3da12971654d9abc2ee8840c4f1592 (table default.Table_20308edcc9354ee4893ae464106f8702 [id=21df672a819243f7a3d16bd6d0a599ac]). Expected version 3 got 0
I20250626 19:57:26.899467 4321 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet a4eb7f65e09642e3960e8672da517e1d (table default.Table_14cb4703932a4df9864af43b426af3fb [id=e970a60e4962403c9f0559ed5dd46115]). Expected version 2 got 0
I20250626 19:57:26.900302 4321 catalog_manager.cc:5582] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:26.902873 4321 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 3cc96a3e9c174cbca0196c5a1b27455c (table default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]). Expected version 1 got 0
I20250626 19:57:26.907649 4654 tablet_service.cc:1468] Processing CreateTablet for tablet 78a8f494f8614e7ebb8ca880b6c80e78 (DEFAULT_TABLE table=default.Table_cc421ddfd27f48b2a8a3bf433f8720f8 [id=eec0405794114b30877bb326de9f9076]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.908727 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a8f494f8614e7ebb8ca880b6c80e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.911003 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet 1944bb9417a741e4adde9422ef5b8222 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:51044
I20250626 19:57:26.910895 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 78a8f494f8614e7ebb8ca880b6c80e78 (DEFAULT_TABLE table=default.Table_cc421ddfd27f48b2a8a3bf433f8720f8 [id=eec0405794114b30877bb326de9f9076]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:26.911963 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a8f494f8614e7ebb8ca880b6c80e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:26.919574 5111 tablet_replica.cc:331] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:26.920228 5111 raft_consensus.cc:2241] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:26.920672 5111 raft_consensus.cc:2270] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:26.922416 5111 ts_tablet_manager.cc:1905] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:26.933861 5154 tablet.cc:1717] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: Skipping requested alter to schema version 3, tablet already version 3
W20250626 19:57:26.937011 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 133963us
I20250626 19:57:26.894477 5092 raft_consensus.cc:357] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:26.943122 5092 raft_consensus.cc:383] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:26.943425 5092 raft_consensus.cc:738] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:26.944386 5092 consensus_queue.cc:260] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:26.967953 5092 ts_tablet_manager.cc:1428] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.076s user 0.006s sys 0.000s
I20250626 19:57:26.969045 5146 raft_consensus.cc:491] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:26.969534 5146 raft_consensus.cc:513] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:26.971341 5146 leader_election.cc:290] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
W20250626 19:57:26.974149 4755 leader_election.cc:343] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Not found: Tablet not found: 612f87796cfa44ddbb19b02797031db9
I20250626 19:57:26.979796 4319 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_d92eac6ec7f04298acf37bef4ed302e4" } new_table_name: "default.Table_87f11a0b0a984eccb7c82abd1984bb4d" modify_external_catalogs: true
I20250626 19:57:26.980051 4320 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_edd247d438724bad8e77a745c85ac645"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:26.981683 4320 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edd247d438724bad8e77a745c85ac645 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:26.984006 5111 ts_tablet_manager.cc:1918] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:26.984385 5111 log.cc:1199] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/1944bb9417a741e4adde9422ef5b8222
I20250626 19:57:26.985026 5111 ts_tablet_manager.cc:1939] T 1944bb9417a741e4adde9422ef5b8222 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:26.987254 4317 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Sending DeleteTablet for 3 replicas of tablet 512dc911bfce49b29c6fc8cd90523723
I20250626 19:57:26.987495 4310 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 1944bb9417a741e4adde9422ef5b8222 (table default.Table_8694e21f21614dc39a47d853c613c2cd [id=dcdd5d68fb8148aabc08e737a2cbabef]) successfully deleted
I20250626 19:57:26.990518 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet 512dc911bfce49b29c6fc8cd90523723 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:56472
I20250626 19:57:26.990658 5092 tablet_bootstrap.cc:492] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
W20250626 19:57:26.992050 4309 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 512dc911bfce49b29c6fc8cd90523723 because the tablet was not found. No further retry: Not found: Tablet not found: 512dc911bfce49b29c6fc8cd90523723
I20250626 19:57:26.996047 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet 512dc911bfce49b29c6fc8cd90523723 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:51044
W20250626 19:57:27.000921 4310 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 512dc911bfce49b29c6fc8cd90523723 because the tablet was not found. No further retry: Not found: Tablet not found: 512dc911bfce49b29c6fc8cd90523723
I20250626 19:57:27.003361 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 512dc911bfce49b29c6fc8cd90523723 (DEFAULT_TABLE table=default.Table_89c4b3541e7c4b08b25f28abf471d42d [id=f0d69891ab4745dbb68ea3c529d1a107]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.003947 4657 tablet_service.cc:1468] Processing CreateTablet for tablet 512dc911bfce49b29c6fc8cd90523723 (DEFAULT_TABLE table=default.Table_89c4b3541e7c4b08b25f28abf471d42d [id=f0d69891ab4745dbb68ea3c529d1a107]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.004375 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 512dc911bfce49b29c6fc8cd90523723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.004951 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 512dc911bfce49b29c6fc8cd90523723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.017114 5092 tablet_bootstrap.cc:654] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:27.021788 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "612f87796cfa44ddbb19b02797031db9" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:27.024300 4756 leader_election.cc:343] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:27.025166 4756 leader_election.cc:304] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
W20250626 19:57:27.027629 4309 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 78a8f494f8614e7ebb8ca880b6c80e78 (table default.Table_cc421ddfd27f48b2a8a3bf433f8720f8 [id=eec0405794114b30877bb326de9f9076], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 78a8f494f8614e7ebb8ca880b6c80e78 consensus state has no leader
I20250626 19:57:27.042167 4798 tablet_service.cc:1515] Processing DeleteTablet for tablet 512dc911bfce49b29c6fc8cd90523723 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:36858
I20250626 19:57:27.039830 5153 raft_consensus.cc:2747] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:27.053509 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 78a8f494f8614e7ebb8ca880b6c80e78 (DEFAULT_TABLE table=default.Table_cc421ddfd27f48b2a8a3bf433f8720f8 [id=eec0405794114b30877bb326de9f9076]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.054889 5091 raft_consensus.cc:357] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.056937 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a8f494f8614e7ebb8ca880b6c80e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.062095 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 512dc911bfce49b29c6fc8cd90523723 (DEFAULT_TABLE table=default.Table_89c4b3541e7c4b08b25f28abf471d42d [id=f0d69891ab4745dbb68ea3c529d1a107]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.063385 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 512dc911bfce49b29c6fc8cd90523723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.068186 4322 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Sending DeleteTablet for 3 replicas of tablet afd1fafb53434957b002fad5c77431d4
I20250626 19:57:27.071333 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet afd1fafb53434957b002fad5c77431d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:51044
W20250626 19:57:27.073383 4310 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet afd1fafb53434957b002fad5c77431d4 because the tablet was not found. No further retry: Not found: Tablet not found: afd1fafb53434957b002fad5c77431d4
I20250626 19:57:27.074406 4317 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_a7a7a4000e8c4ebb91954cd1343dd235"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:27.075500 4315 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_689b779650c544c7baa707961e76e953"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:27.077291 4315 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_689b779650c544c7baa707961e76e953 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:57:27.077361 4317 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7a7a4000e8c4ebb91954cd1343dd235 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:27.082954 5091 raft_consensus.cc:383] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:27.083393 5091 raft_consensus.cc:738] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:27.084362 5091 consensus_queue.cc:260] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.096434 4322 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_443891de930144a484281a0abf7d60f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:27.099185 4322 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_443891de930144a484281a0abf7d60f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:27.111935 4318 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet a4eb7f65e09642e3960e8672da517e1d (table default.Table_f24d9839a6b84ea691c394b6b3ddf3b3 [id=e970a60e4962403c9f0559ed5dd46115]). Expected version 3 got 2
I20250626 19:57:27.112668 4318 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 3cc96a3e9c174cbca0196c5a1b27455c (table default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]). Expected version 1 got 0
I20250626 19:57:27.115909 4316 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet a4eb7f65e09642e3960e8672da517e1d (table default.Table_f24d9839a6b84ea691c394b6b3ddf3b3 [id=e970a60e4962403c9f0559ed5dd46115]). Expected version 3 got 2
I20250626 19:57:27.122272 4321 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_ef500e9a9c0c4ef69cb8f1905015a3f1" } new_table_name: "default.Table_b401c8a5ab6744029f693b1b5e0f306d" modify_external_catalogs: true
I20250626 19:57:27.133401 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 612f87796cfa44ddbb19b02797031db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:36858
I20250626 19:57:27.134402 5136 tablet.cc:1722] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250626 19:57:27.139895 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 115712us
I20250626 19:57:27.143229 5154 tablet.cc:1722] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250626 19:57:27.148530 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 499f6ae3262940b4859c14a53722482f (DEFAULT_TABLE table=default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.149595 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 499f6ae3262940b4859c14a53722482f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.151120 4800 tablet_service.cc:1515] Processing DeleteTablet for tablet afd1fafb53434957b002fad5c77431d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:36858
I20250626 19:57:27.152412 4954 tablet_service.cc:1468] Processing CreateTablet for tablet afd1fafb53434957b002fad5c77431d4 (DEFAULT_TABLE table=default.Table_b6e3c1f4224b4dee8c044d2548276942 [id=cb6e1a290c9c47c0bef9de879ab06bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.153079 5162 tablet_replica.cc:331] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:27.153458 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd1fafb53434957b002fad5c77431d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.153867 5162 raft_consensus.cc:2241] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:27.154377 5162 raft_consensus.cc:2270] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:27.158197 5089 tablet_bootstrap.cc:492] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:27.158596 5089 ts_tablet_manager.cc:1397] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.404s user 0.008s sys 0.008s
I20250626 19:57:27.164395 5092 tablet_bootstrap.cc:492] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:27.164806 5092 ts_tablet_manager.cc:1397] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.174s user 0.045s sys 0.031s
I20250626 19:57:27.168228 5091 ts_tablet_manager.cc:1428] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.310s user 0.007s sys 0.000s
W20250626 19:57:27.169437 4308 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 512dc911bfce49b29c6fc8cd90523723 because the tablet was not found. No further retry: Not found: Tablet not found: 512dc911bfce49b29c6fc8cd90523723
I20250626 19:57:27.171066 4801 tablet_service.cc:1468] Processing CreateTablet for tablet afd1fafb53434957b002fad5c77431d4 (DEFAULT_TABLE table=default.Table_b6e3c1f4224b4dee8c044d2548276942 [id=cb6e1a290c9c47c0bef9de879ab06bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.172155 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd1fafb53434957b002fad5c77431d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.175987 5091 tablet_bootstrap.cc:492] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:27.184705 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 499f6ae3262940b4859c14a53722482f (DEFAULT_TABLE table=default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.185801 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 499f6ae3262940b4859c14a53722482f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.196813 5162 ts_tablet_manager.cc:1905] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:27.197563 4316 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_603f3fe8b7224f19b395762b57f2fa5a" } modify_external_catalogs: true
I20250626 19:57:27.198135 4316 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_603f3fe8b7224f19b395762b57f2fa5a" } modify_external_catalogs: true
I20250626 19:57:27.201138 4652 tablet_service.cc:1468] Processing CreateTablet for tablet afd1fafb53434957b002fad5c77431d4 (DEFAULT_TABLE table=default.Table_b6e3c1f4224b4dee8c044d2548276942 [id=cb6e1a290c9c47c0bef9de879ab06bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.202739 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd1fafb53434957b002fad5c77431d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.203007 5092 raft_consensus.cc:357] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.203655 5092 raft_consensus.cc:383] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:27.203918 5092 raft_consensus.cc:738] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:27.204494 5092 consensus_queue.cc:260] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.206475 5092 ts_tablet_manager.cc:1428] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.041s user 0.003s sys 0.000s
I20250626 19:57:27.160836 5089 raft_consensus.cc:357] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.207052 5089 raft_consensus.cc:383] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:27.207312 5089 raft_consensus.cc:738] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:27.207894 5089 consensus_queue.cc:260] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.209828 5089 ts_tablet_manager.cc:1428] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.051s user 0.003s sys 0.000s
I20250626 19:57:27.210501 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet 612f87796cfa44ddbb19b02797031db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:56472
W20250626 19:57:27.210728 4319 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1cb6bda529624ac5b5a87a03b4a02d56 has no consensus state
I20250626 19:57:27.219564 5092 tablet_bootstrap.cc:492] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:27.220255 4655 tablet_service.cc:1515] Processing DeleteTablet for tablet afd1fafb53434957b002fad5c77431d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:56472
I20250626 19:57:27.221769 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 499f6ae3262940b4859c14a53722482f (DEFAULT_TABLE table=default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.222980 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 499f6ae3262940b4859c14a53722482f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.223475 5089 tablet_bootstrap.cc:492] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:27.230322 5092 tablet_bootstrap.cc:654] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:27.273640 4323 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_f24d9839a6b84ea691c394b6b3ddf3b3" } modify_external_catalogs: true
I20250626 19:57:27.274225 4323 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_f24d9839a6b84ea691c394b6b3ddf3b3" } modify_external_catalogs: true
W20250626 19:57:27.274567 4309 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 612f87796cfa44ddbb19b02797031db9 because the tablet was not found. No further retry: Not found: Tablet not found: 612f87796cfa44ddbb19b02797031db9
W20250626 19:57:27.275909 4309 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet afd1fafb53434957b002fad5c77431d4 because the tablet was not found. No further retry: Not found: Tablet not found: afd1fafb53434957b002fad5c77431d4
I20250626 19:57:27.282894 5091 tablet_bootstrap.cc:654] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:27.294101 5089 tablet_bootstrap.cc:654] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:27.304303 5160 raft_consensus.cc:491] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:27.304774 5160 raft_consensus.cc:513] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.283615 5135 tablet.cc:1722] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250626 19:57:27.306644 5160 leader_election.cc:290] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:27.307325 5160 raft_consensus.cc:491] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:27.307718 5160 raft_consensus.cc:513] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.309386 5160 leader_election.cc:290] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:27.311517 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd0ca2fab40f45e5a1f9bd6059ca1f08" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:27.312809 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5785508e37624f02bc2f6cc93007e3f8" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:27.313364 4677 raft_consensus.cc:2466] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:27.313774 5162 ts_tablet_manager.cc:1918] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:27.314193 5162 log.cc:1199] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/612f87796cfa44ddbb19b02797031db9
I20250626 19:57:27.315055 5162 ts_tablet_manager.cc:1939] T 612f87796cfa44ddbb19b02797031db9 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
W20250626 19:57:27.315553 4755 leader_election.cc:343] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:27.316166 4755 leader_election.cc:304] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:27.316921 5153 raft_consensus.cc:2802] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:27.317219 5153 raft_consensus.cc:491] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:27.317503 5153 raft_consensus.cc:3058] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:27.318125 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd0ca2fab40f45e5a1f9bd6059ca1f08" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:27.318912 4977 raft_consensus.cc:2466] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:27.321058 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5785508e37624f02bc2f6cc93007e3f8" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:27.321712 4977 raft_consensus.cc:2466] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:27.323233 4756 leader_election.cc:304] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:27.324203 5160 raft_consensus.cc:2802] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:27.324524 5160 raft_consensus.cc:491] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:27.324865 5160 raft_consensus.cc:3058] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:27.331136 4308 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 612f87796cfa44ddbb19b02797031db9 (table default.Table_c5ff61c31334492c9cc8dede96094236 [id=23c193073c364d7c921797f0940caf10]) successfully deleted
I20250626 19:57:27.331290 5153 raft_consensus.cc:513] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.331682 5160 raft_consensus.cc:513] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:27.332592 4308 catalog_manager.cc:4908] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet afd1fafb53434957b002fad5c77431d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afd1fafb53434957b002fad5c77431d4 already in progress: creating tablet
I20250626 19:57:27.334403 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5785508e37624f02bc2f6cc93007e3f8" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:27.334969 4677 raft_consensus.cc:3058] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:27.337005 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd0ca2fab40f45e5a1f9bd6059ca1f08" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
W20250626 19:57:27.338225 4755 leader_election.cc:343] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:27.338702 5160 leader_election.cc:290] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:27.339811 5153 leader_election.cc:290] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:27.340921 5089 tablet_bootstrap.cc:492] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:27.343094 5089 ts_tablet_manager.cc:1397] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.120s user 0.010s sys 0.004s
I20250626 19:57:27.345427 5089 raft_consensus.cc:357] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.346143 5089 raft_consensus.cc:383] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:27.346443 5089 raft_consensus.cc:738] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:27.347085 5089 consensus_queue.cc:260] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.349045 5089 ts_tablet_manager.cc:1428] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250626 19:57:27.349905 5089 tablet_bootstrap.cc:492] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:27.351821 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5785508e37624f02bc2f6cc93007e3f8" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:27.352542 4977 raft_consensus.cc:3058] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:27.354051 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd0ca2fab40f45e5a1f9bd6059ca1f08" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:27.354600 4979 raft_consensus.cc:3058] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:27.362646 5096 consensus_queue.cc:1035] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:27.366933 4977 raft_consensus.cc:2466] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:27.368353 4979 raft_consensus.cc:2466] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:27.381592 5096 consensus_queue.cc:1035] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:27.409124 4677 raft_consensus.cc:2466] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:27.419719 4322 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_20308edcc9354ee4893ae464106f8702" } new_table_name: "default.Table_cdb0c85507da4162961044be294061a7" modify_external_catalogs: true
I20250626 19:57:27.379634 4756 leader_election.cc:304] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
W20250626 19:57:27.436234 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 56749.5us
I20250626 19:57:27.437094 4756 leader_election.cc:304] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:27.438922 5153 raft_consensus.cc:2802] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:27.439695 5153 raft_consensus.cc:695] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:27.440729 5153 consensus_queue.cc:237] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.442574 5091 tablet_bootstrap.cc:492] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:27.467381 4317 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_f579d2c76b104fc88466bcdcd54d2603"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:27.469069 4317 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f579d2c76b104fc88466bcdcd54d2603 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:27.474772 5091 ts_tablet_manager.cc:1397] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.299s user 0.005s sys 0.006s
I20250626 19:57:27.438839 5160 raft_consensus.cc:2802] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:27.479580 5160 raft_consensus.cc:695] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:27.480018 4316 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet dd34eb96a2f54585b5f40d962d3640a1
I20250626 19:57:27.480382 5160 consensus_queue.cc:237] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.493597 5092 tablet_bootstrap.cc:492] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:27.494076 5092 ts_tablet_manager.cc:1397] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.275s user 0.094s sys 0.069s
W20250626 19:57:27.495334 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 64796.8us
I20250626 19:57:27.499231 5092 raft_consensus.cc:357] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.499950 5092 raft_consensus.cc:383] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:27.500228 5092 raft_consensus.cc:738] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:27.501039 5092 consensus_queue.cc:260] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.512519 4316 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_e25c8c1da59041118a0072aa639e3771"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:27.514281 4316 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e25c8c1da59041118a0072aa639e3771 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:27.513834 5092 ts_tablet_manager.cc:1428] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.019s user 0.002s sys 0.003s
I20250626 19:57:27.524660 5092 tablet_bootstrap.cc:492] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:27.527547 5091 raft_consensus.cc:357] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.537643 5092 tablet_bootstrap.cc:654] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:27.569816 5089 tablet_bootstrap.cc:654] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:27.570108 5135 tablet.cc:1722] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250626 19:57:27.572270 5091 raft_consensus.cc:383] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:27.572651 5091 raft_consensus.cc:738] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:27.577139 5091 consensus_queue.cc:260] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.584096 4322 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_01a5eaed994e4495bd80c775e1a081a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:27.586585 4322 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01a5eaed994e4495bd80c775e1a081a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:27.587738 5091 ts_tablet_manager.cc:1428] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.113s user 0.005s sys 0.000s
I20250626 19:57:27.601118 5091 tablet_bootstrap.cc:492] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:27.604254 4315 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_d23ee99af7b845b78d2200b962f681a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:27.605963 4315 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d23ee99af7b845b78d2200b962f681a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:27.600358 5154 tablet.cc:1722] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250626 19:57:27.609582 5136 tablet.cc:1722] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250626 19:57:27.643321 5089 tablet_bootstrap.cc:492] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:27.646723 5089 ts_tablet_manager.cc:1397] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.297s user 0.008s sys 0.002s
I20250626 19:57:27.649567 5089 raft_consensus.cc:357] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.650362 5089 raft_consensus.cc:383] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:27.650627 5089 raft_consensus.cc:738] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:27.657482 5089 consensus_queue.cc:260] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:27.661698 4321 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5785508e37624f02bc2f6cc93007e3f8 consensus state has no leader
I20250626 19:57:27.661348 4652 tablet_service.cc:1468] Processing CreateTablet for tablet 9f91a78f6d474133976242ea8d259e29 (DEFAULT_TABLE table=default.Table_b2fa45f3c65e4e56b7617277fe9f16ca [id=048633318f5c4de0bd3a11f63c3f8103]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.662149 4954 tablet_service.cc:1468] Processing CreateTablet for tablet 9f91a78f6d474133976242ea8d259e29 (DEFAULT_TABLE table=default.Table_b2fa45f3c65e4e56b7617277fe9f16ca [id=048633318f5c4de0bd3a11f63c3f8103]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.662585 4323 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Sending DeleteTablet for 3 replicas of tablet a4eb7f65e09642e3960e8672da517e1d
I20250626 19:57:27.663058 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f91a78f6d474133976242ea8d259e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.664995 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 4d8ffa9b658c413fa06d02ee6635dd1b (DEFAULT_TABLE table=default.Table_2ab026a162784703bf424759f2114968 [id=4419bc765f3a4ff1a104bb656bf642ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.665601 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f91a78f6d474133976242ea8d259e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.666018 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8ffa9b658c413fa06d02ee6635dd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.666824 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 4d8ffa9b658c413fa06d02ee6635dd1b (DEFAULT_TABLE table=default.Table_2ab026a162784703bf424759f2114968 [id=4419bc765f3a4ff1a104bb656bf642ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.667763 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8ffa9b658c413fa06d02ee6635dd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.669399 5091 tablet_bootstrap.cc:654] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:27.671414 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 9f91a78f6d474133976242ea8d259e29 (DEFAULT_TABLE table=default.Table_b2fa45f3c65e4e56b7617277fe9f16ca [id=048633318f5c4de0bd3a11f63c3f8103]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.672610 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f91a78f6d474133976242ea8d259e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.674728 5092 tablet_bootstrap.cc:492] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:27.675146 5092 ts_tablet_manager.cc:1397] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.151s user 0.008s sys 0.002s
I20250626 19:57:27.678242 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet a4eb7f65e09642e3960e8672da517e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:36858
I20250626 19:57:27.679020 5162 tablet_replica.cc:331] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:27.679658 4654 tablet_service.cc:1515] Processing DeleteTablet for tablet a4eb7f65e09642e3960e8672da517e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:56472
I20250626 19:57:27.679729 5162 raft_consensus.cc:2241] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:27.680455 5112 tablet_replica.cc:331] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:27.680658 5162 raft_consensus.cc:2270] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:27.681175 5112 raft_consensus.cc:2241] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:27.681895 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet a4eb7f65e09642e3960e8672da517e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:51044
I20250626 19:57:27.682246 5112 raft_consensus.cc:2270] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:27.684401 5162 ts_tablet_manager.cc:1905] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:27.684980 5112 ts_tablet_manager.cc:1905] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:27.685384 4318 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_cc421ddfd27f48b2a8a3bf433f8720f8" } modify_external_catalogs: true
I20250626 19:57:27.685962 4318 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_cc421ddfd27f48b2a8a3bf433f8720f8" } modify_external_catalogs: true
I20250626 19:57:27.690429 4957 tablet_service.cc:1468] Processing CreateTablet for tablet 4b3bed19e7e34d24b995840a3bde5ef0 (DEFAULT_TABLE table=default.Table_ad656f771f554bc98b2dc4db559e3acb [id=3c86bc87a4a64fd8af0af50c516a2242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.693545 4956 tablet_service.cc:1468] Processing CreateTablet for tablet 1cb6bda529624ac5b5a87a03b4a02d56 (DEFAULT_TABLE table=default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.696121 4657 tablet_service.cc:1468] Processing CreateTablet for tablet 1cb6bda529624ac5b5a87a03b4a02d56 (DEFAULT_TABLE table=default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.691291 4798 tablet_service.cc:1468] Processing CreateTablet for tablet 4b3bed19e7e34d24b995840a3bde5ef0 (DEFAULT_TABLE table=default.Table_ad656f771f554bc98b2dc4db559e3acb [id=3c86bc87a4a64fd8af0af50c516a2242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.697757 4798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b3bed19e7e34d24b995840a3bde5ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.697613 4654 tablet_service.cc:1468] Processing CreateTablet for tablet 4b3bed19e7e34d24b995840a3bde5ef0 (DEFAULT_TABLE table=default.Table_ad656f771f554bc98b2dc4db559e3acb [id=3c86bc87a4a64fd8af0af50c516a2242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.699242 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cb6bda529624ac5b5a87a03b4a02d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.692430 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 4d8ffa9b658c413fa06d02ee6635dd1b (DEFAULT_TABLE table=default.Table_2ab026a162784703bf424759f2114968 [id=4419bc765f3a4ff1a104bb656bf642ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.703960 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b3bed19e7e34d24b995840a3bde5ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.704078 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8ffa9b658c413fa06d02ee6635dd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.714783 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b3bed19e7e34d24b995840a3bde5ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.715898 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 1cb6bda529624ac5b5a87a03b4a02d56 (DEFAULT_TABLE table=default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:27.717511 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cb6bda529624ac5b5a87a03b4a02d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.721621 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cb6bda529624ac5b5a87a03b4a02d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:27.734030 5089 ts_tablet_manager.cc:1428] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.087s user 0.002s sys 0.004s
I20250626 19:57:27.737267 5089 tablet_bootstrap.cc:492] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:27.706105 4323 catalog_manager.cc:5582] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:27.744289 4323 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 3cc96a3e9c174cbca0196c5a1b27455c (table default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]). Expected version 1 got 0
I20250626 19:57:27.745481 4323 catalog_manager.cc:5582] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:27.751125 4323 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 5785508e37624f02bc2f6cc93007e3f8 (table default.Table_b401c8a5ab6744029f693b1b5e0f306d [id=95832380520e4cf18d762654fb684db3]). Expected version 1 got 0
I20250626 19:57:27.755162 4323 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 3bd8dae5699c4725915814044d76a257 (table default.Table_d867f59f19a048b6a533acb0236b876b [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 1 got 0
I20250626 19:57:27.756618 5092 raft_consensus.cc:357] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.757341 5092 raft_consensus.cc:383] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:27.757730 5092 raft_consensus.cc:738] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:27.758523 5092 consensus_queue.cc:260] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.776968 5091 tablet_bootstrap.cc:492] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:27.777441 5091 ts_tablet_manager.cc:1397] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.177s user 0.005s sys 0.007s
I20250626 19:57:27.760057 4322 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_edd247d438724bad8e77a745c85ac645" } new_table_name: "default.Table_27bcd169607a45599dc5a644e6ea5dd7" modify_external_catalogs: true
I20250626 19:57:27.784355 5091 raft_consensus.cc:357] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.784978 5091 raft_consensus.cc:383] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:27.785247 5091 raft_consensus.cc:738] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:27.785954 5091 consensus_queue.cc:260] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.795099 5092 ts_tablet_manager.cc:1428] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.120s user 0.007s sys 0.000s
I20250626 19:57:27.742668 5089 tablet_bootstrap.cc:654] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:27.805922 5092 tablet_bootstrap.cc:492] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:27.812978 5091 ts_tablet_manager.cc:1428] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.035s user 0.003s sys 0.002s
I20250626 19:57:27.790438 5112 ts_tablet_manager.cc:1918] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250626 19:57:27.813597 5112 log.cc:1199] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/a4eb7f65e09642e3960e8672da517e1d
I20250626 19:57:27.814743 5112 ts_tablet_manager.cc:1939] T a4eb7f65e09642e3960e8672da517e1d P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:27.818269 5091 tablet_bootstrap.cc:492] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:27.823935 4309 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet a4eb7f65e09642e3960e8672da517e1d (table default.Table_f24d9839a6b84ea691c394b6b3ddf3b3 [id=e970a60e4962403c9f0559ed5dd46115]) successfully deleted
I20250626 19:57:27.828941 4318 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Sending DeleteTablet for 3 replicas of tablet 78a8f494f8614e7ebb8ca880b6c80e78
I20250626 19:57:27.831979 4800 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a8f494f8614e7ebb8ca880b6c80e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:36858
I20250626 19:57:27.831537 4654 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a8f494f8614e7ebb8ca880b6c80e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:56472
I20250626 19:57:27.832901 5176 raft_consensus.cc:491] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:27.833317 5176 raft_consensus.cc:513] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.835340 5176 leader_election.cc:290] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:27.836757 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa93fa6c652f4aca848e1e4910fb8175" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:27.837350 5180 tablet_replica.cc:331] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:27.837509 5089 tablet_bootstrap.cc:492] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:27.837990 5089 ts_tablet_manager.cc:1397] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.101s user 0.011s sys 0.001s
I20250626 19:57:27.837352 4821 raft_consensus.cc:2466] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:27.838407 5180 raft_consensus.cc:2241] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:27.839136 4609 leader_election.cc:304] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:27.839797 5180 raft_consensus.cc:2270] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:27.840695 4315 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_65941c5740784059b05dacd5e2f1a5c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:27.842372 4315 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65941c5740784059b05dacd5e2f1a5c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:27.845851 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa93fa6c652f4aca848e1e4910fb8175" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:27.845822 5089 raft_consensus.cc:357] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.846449 5089 raft_consensus.cc:383] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:27.846431 4978 raft_consensus.cc:2466] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:27.848380 5180 ts_tablet_manager.cc:1905] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:27.852151 5176 raft_consensus.cc:2802] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:27.852512 5176 raft_consensus.cc:491] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:27.852820 5176 raft_consensus.cc:3058] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:27.848858 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a8f494f8614e7ebb8ca880b6c80e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:51044
I20250626 19:57:27.846699 5089 raft_consensus.cc:738] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:27.859395 5089 consensus_queue.cc:260] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.850819 5092 tablet_bootstrap.cc:654] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:27.863241 4309 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 78a8f494f8614e7ebb8ca880b6c80e78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78a8f494f8614e7ebb8ca880b6c80e78 already in progress: creating tablet
I20250626 19:57:27.865769 5162 ts_tablet_manager.cc:1918] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
W20250626 19:57:27.868359 4322 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44960f8db08942879cde6f96e9c620da has no consensus state
I20250626 19:57:27.870885 5162 log.cc:1199] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/a4eb7f65e09642e3960e8672da517e1d
I20250626 19:57:27.871811 5162 ts_tablet_manager.cc:1939] T a4eb7f65e09642e3960e8672da517e1d P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:27.874191 4308 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet a4eb7f65e09642e3960e8672da517e1d (table default.Table_f24d9839a6b84ea691c394b6b3ddf3b3 [id=e970a60e4962403c9f0559ed5dd46115]) successfully deleted
I20250626 19:57:27.874486 5089 ts_tablet_manager.cc:1428] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20250626 19:57:27.875893 5091 tablet_bootstrap.cc:654] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:27.878799 5176 raft_consensus.cc:513] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.881780 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa93fa6c652f4aca848e1e4910fb8175" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:27.882330 4978 raft_consensus.cc:3058] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:27.883437 4308 catalog_manager.cc:4908] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 78a8f494f8614e7ebb8ca880b6c80e78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78a8f494f8614e7ebb8ca880b6c80e78 already in progress: creating tablet
I20250626 19:57:27.884405 5092 tablet_bootstrap.cc:492] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:27.884781 5092 ts_tablet_manager.cc:1397] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.079s user 0.009s sys 0.001s
I20250626 19:57:27.887012 5092 raft_consensus.cc:357] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.887463 5089 tablet_bootstrap.cc:492] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:27.887605 5092 raft_consensus.cc:383] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:27.887815 5092 raft_consensus.cc:738] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:27.891950 5092 consensus_queue.cc:260] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.901274 4322 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_cdb0c85507da4162961044be294061a7" } new_table_name: "default.Table_1a93e36798c7434da2c9cd04201320cd" modify_external_catalogs: true
I20250626 19:57:27.902047 4318 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_a7a7a4000e8c4ebb91954cd1343dd235" } modify_external_catalogs: true
I20250626 19:57:27.902683 4318 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_a7a7a4000e8c4ebb91954cd1343dd235" } modify_external_catalogs: true
I20250626 19:57:27.907682 5168 raft_consensus.cc:491] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:27.907708 5092 ts_tablet_manager.cc:1428] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.023s user 0.006s sys 0.001s
I20250626 19:57:27.908171 5168 raft_consensus.cc:513] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.910174 5168 leader_election.cc:290] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:27.910416 5089 tablet_bootstrap.cc:654] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:27.911859 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bd8dae5699c4725915814044d76a257" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:27.912503 4821 raft_consensus.cc:2466] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:27.914248 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bd8dae5699c4725915814044d76a257" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:27.914808 5092 tablet_bootstrap.cc:492] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:27.915812 4609 leader_election.cc:304] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
W20250626 19:57:27.916786 4611 leader_election.cc:343] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:27.918053 5168 raft_consensus.cc:2802] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:27.918417 5168 raft_consensus.cc:491] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:27.918913 5180 ts_tablet_manager.cc:1918] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250626 19:57:27.918846 5168 raft_consensus.cc:3058] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:27.919418 5180 log.cc:1199] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/a4eb7f65e09642e3960e8672da517e1d
I20250626 19:57:27.921525 5176 leader_election.cc:290] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:27.922046 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa93fa6c652f4aca848e1e4910fb8175" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:27.912032 4978 raft_consensus.cc:2466] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:27.922592 4821 raft_consensus.cc:3058] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:27.922935 5180 ts_tablet_manager.cc:1939] T a4eb7f65e09642e3960e8672da517e1d P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:27.926662 5092 tablet_bootstrap.cc:654] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:27.927979 4611 leader_election.cc:304] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:27.931111 4310 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet a4eb7f65e09642e3960e8672da517e1d (table default.Table_f24d9839a6b84ea691c394b6b3ddf3b3 [id=e970a60e4962403c9f0559ed5dd46115]) successfully deleted
W20250626 19:57:27.931787 4310 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 78a8f494f8614e7ebb8ca880b6c80e78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78a8f494f8614e7ebb8ca880b6c80e78 already in progress: creating tablet
I20250626 19:57:27.937300 4821 raft_consensus.cc:2466] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:27.941112 5091 tablet_bootstrap.cc:492] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:27.941560 5091 ts_tablet_manager.cc:1397] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.123s user 0.008s sys 0.000s
I20250626 19:57:27.947029 5176 raft_consensus.cc:2802] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:27.947511 5176 raft_consensus.cc:695] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:27.948297 5176 consensus_queue.cc:237] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.953723 5168 raft_consensus.cc:513] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.956460 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bd8dae5699c4725915814044d76a257" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:27.957111 5089 tablet_bootstrap.cc:492] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:27.957621 5089 ts_tablet_manager.cc:1397] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.001s
W20250626 19:57:27.957891 4611 leader_election.cc:343] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:27.959586 5168 leader_election.cc:290] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:27.959774 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bd8dae5699c4725915814044d76a257" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:27.960337 4821 raft_consensus.cc:3058] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:27.960783 5089 raft_consensus.cc:357] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.961629 5089 raft_consensus.cc:383] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:27.970221 5089 raft_consensus.cc:738] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:27.977954 4316 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 3bd8dae5699c4725915814044d76a257 (table default.Table_d867f59f19a048b6a533acb0236b876b [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 1 got 0
I20250626 19:57:27.980150 4316 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 3cc96a3e9c174cbca0196c5a1b27455c (table default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]). Expected version 1 got 0
I20250626 19:57:27.982865 5091 raft_consensus.cc:357] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.987612 5091 raft_consensus.cc:383] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:27.987865 5091 raft_consensus.cc:738] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:27.988884 4677 raft_consensus.cc:1273] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Refusing update from remote peer 93f885c6802341c889a40e9c20fbfff1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:57:27.989005 4978 raft_consensus.cc:1273] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Refusing update from remote peer 93f885c6802341c889a40e9c20fbfff1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:57:27.989217 5091 consensus_queue.cc:260] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:27.991880 5091 ts_tablet_manager.cc:1428] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.050s user 0.008s sys 0.000s
I20250626 19:57:27.993407 5091 tablet_bootstrap.cc:492] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:27.994917 4320 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 3cc96a3e9c174cbca0196c5a1b27455c (table default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]). Expected version 1 got 0
I20250626 19:57:27.972568 4821 raft_consensus.cc:2466] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:27.990325 5160 consensus_queue.cc:1035] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:28.003855 5089 consensus_queue.cc:260] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.004158 4609 leader_election.cc:304] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:28.005618 5168 raft_consensus.cc:2802] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:28.006508 5168 raft_consensus.cc:695] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:28.007557 5168 consensus_queue.cc:237] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.016032 4315 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_beec4e8c60ae4e6880c0ff709d3c0ff3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:28.017778 4315 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_beec4e8c60ae4e6880c0ff709d3c0ff3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:28.022523 4321 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_bd7be9809f0d462380d65cb6ed3f8c1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:28.023833 5153 consensus_queue.cc:1035] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:57:28.024170 4321 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd7be9809f0d462380d65cb6ed3f8c1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:28.022085 4319 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_fdb231070332470b994fc496021fde25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:28.034037 4319 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdb231070332470b994fc496021fde25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:28.043469 4317 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_b401c8a5ab6744029f693b1b5e0f306d" } modify_external_catalogs: true
I20250626 19:57:28.044087 4317 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_b401c8a5ab6744029f693b1b5e0f306d" } modify_external_catalogs: true
I20250626 19:57:28.056216 5092 tablet_bootstrap.cc:492] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:28.056694 5092 ts_tablet_manager.cc:1397] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.142s user 0.042s sys 0.020s
W20250626 19:57:28.059659 4308 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 44960f8db08942879cde6f96e9c620da (table default.Table_27bcd169607a45599dc5a644e6ea5dd7 [id=8df233b9523c49adb3d24727bda24b9e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 44960f8db08942879cde6f96e9c620da has no consensus state
I20250626 19:57:28.064704 5089 ts_tablet_manager.cc:1428] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.107s user 0.006s sys 0.010s
I20250626 19:57:28.072598 4324 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_d867f59f19a048b6a533acb0236b876b" } new_table_name: "default.Table_94a9ea25bac540f59bd2e2b6beb127c1" modify_external_catalogs: true
I20250626 19:57:28.074736 5089 tablet_bootstrap.cc:492] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:28.078462 5135 tablet.cc:1722] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:28.081542 5136 tablet.cc:1722] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:28.082502 5089 tablet_bootstrap.cc:654] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.083465 5188 raft_consensus.cc:491] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:28.083876 5188 raft_consensus.cc:513] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.089395 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a8f494f8614e7ebb8ca880b6c80e78" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:28.070478 5092 raft_consensus.cc:357] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:28.090804 4755 leader_election.cc:343] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:28.091713 5188 leader_election.cc:290] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:28.090076 5092 raft_consensus.cc:383] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.093690 5092 raft_consensus.cc:738] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.093860 4318 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 6a5c135a48304843ada5d779fd48cd6a
I20250626 19:57:28.095131 5092 consensus_queue.cc:260] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.107949 5092 ts_tablet_manager.cc:1428] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.051s user 0.002s sys 0.007s
I20250626 19:57:28.131510 5089 tablet_bootstrap.cc:492] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:28.131567 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a8f494f8614e7ebb8ca880b6c80e78" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:28.131464 5136 tablet.cc:1722] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:28.131954 5089 ts_tablet_manager.cc:1397] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.003s
W20250626 19:57:28.136914 4756 leader_election.cc:343] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:28.137409 4756 leader_election.cc:304] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:28.138270 5160 raft_consensus.cc:2747] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:28.147823 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet a4eb7f65e09642e3960e8672da517e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:51044
W20250626 19:57:28.166165 4310 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet a4eb7f65e09642e3960e8672da517e1d because the tablet was not found. No further retry: Not found: Tablet not found: a4eb7f65e09642e3960e8672da517e1d
I20250626 19:57:28.170867 5092 tablet_bootstrap.cc:492] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:28.176210 5092 tablet_bootstrap.cc:654] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.183465 4315 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_579a1ea24d1744068526a0765d5966ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:28.185631 4315 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_579a1ea24d1744068526a0765d5966ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:28.187510 5092 tablet_bootstrap.cc:492] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:28.167783 5089 raft_consensus.cc:357] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.187872 5092 ts_tablet_manager.cc:1397] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20250626 19:57:28.187862 5089 raft_consensus.cc:383] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.188133 5089 raft_consensus.cc:738] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.189008 5089 consensus_queue.cc:260] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.196406 5196 tablet.cc:1722] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:28.205603 5089 ts_tablet_manager.cc:1428] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.073s user 0.005s sys 0.001s
I20250626 19:57:28.213924 5135 tablet.cc:1722] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:28.220870 5089 tablet_bootstrap.cc:492] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:28.190145 5092 raft_consensus.cc:357] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.223892 5092 raft_consensus.cc:383] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.224275 5092 raft_consensus.cc:738] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.224845 5092 consensus_queue.cc:260] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.226927 5092 ts_tablet_manager.cc:1428] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20250626 19:57:28.228128 5198 tablet.cc:1722] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:28.038301 5091 tablet_bootstrap.cc:654] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.250356 5193 tablet.cc:1722] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250626 19:57:28.256347 5089 tablet_bootstrap.cc:654] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.279085 5194 tablet.cc:1722] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250626 19:57:28.279500 4320 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 3bd8dae5699c4725915814044d76a257 (table default.Table_d867f59f19a048b6a533acb0236b876b [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 1 got 0
I20250626 19:57:28.280216 4320 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5785508e37624f02bc2f6cc93007e3f8 (table default.Table_b401c8a5ab6744029f693b1b5e0f306d [id=95832380520e4cf18d762654fb684db3]). Expected version 1 got 0
W20250626 19:57:28.281144 4324 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bd8dae5699c4725915814044d76a257 consensus state has no leader
I20250626 19:57:28.285068 5089 tablet_bootstrap.cc:492] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:28.285434 5089 ts_tablet_manager.cc:1397] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.004s
I20250626 19:57:28.287585 5089 raft_consensus.cc:357] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:28.288209 5089 raft_consensus.cc:383] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.289279 4654 tablet_service.cc:1468] Processing CreateTablet for tablet af58f34e2248432fae5c084bd3388028 (DEFAULT_TABLE table=default.Table_e25c8c1da59041118a0072aa639e3771 [id=bea62f96f1cd4c2ab35704acf7def541]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:28.290313 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af58f34e2248432fae5c084bd3388028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:28.291486 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet a4eb7f65e09642e3960e8672da517e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:56472
I20250626 19:57:28.288445 5089 raft_consensus.cc:738] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
W20250626 19:57:28.297940 4309 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet a4eb7f65e09642e3960e8672da517e1d because the tablet was not found. No further retry: Not found: Tablet not found: a4eb7f65e09642e3960e8672da517e1d
I20250626 19:57:28.299103 5089 consensus_queue.cc:260] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:28.312085 5089 ts_tablet_manager.cc:1428] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20250626 19:57:28.313012 5089 tablet_bootstrap.cc:492] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:28.319134 4798 tablet_service.cc:1468] Processing CreateTablet for tablet af58f34e2248432fae5c084bd3388028 (DEFAULT_TABLE table=default.Table_e25c8c1da59041118a0072aa639e3771 [id=bea62f96f1cd4c2ab35704acf7def541]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:28.320163 4798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af58f34e2248432fae5c084bd3388028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:28.321344 5091 tablet_bootstrap.cc:492] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:28.321744 5091 ts_tablet_manager.cc:1397] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.329s user 0.007s sys 0.005s
I20250626 19:57:28.229589 5092 tablet_bootstrap.cc:492] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:28.242055 5199 tablet.cc:1722] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250626 19:57:28.313923 5148 raft_consensus.cc:491] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:28.328403 5148 raft_consensus.cc:513] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.330240 5148 leader_election.cc:290] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:28.332561 5091 raft_consensus.cc:357] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.333225 5091 raft_consensus.cc:383] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.333468 5091 raft_consensus.cc:738] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.337848 4677 raft_consensus.cc:3058] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:28.347054 5089 tablet_bootstrap.cc:654] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:28.334509 4911 leader_election.cc:343] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Not found: Tablet not found: 612f87796cfa44ddbb19b02797031db9
W20250626 19:57:28.347950 4912 leader_election.cc:343] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Not found: Tablet not found: 612f87796cfa44ddbb19b02797031db9
I20250626 19:57:28.348773 4912 leader_election.cc:304] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:28.351656 4324 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_fdb231070332470b994fc496021fde25" } new_table_name: "default.Table_c98cdc17c95f4f1295e667b2c8324085" modify_external_catalogs: true
I20250626 19:57:28.355029 5148 raft_consensus.cc:2747] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:28.359184 4315 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_65941c5740784059b05dacd5e2f1a5c3" } modify_external_catalogs: true
I20250626 19:57:28.359853 4315 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_65941c5740784059b05dacd5e2f1a5c3" } modify_external_catalogs: true
I20250626 19:57:28.362839 5188 consensus_queue.cc:1035] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:28.364413 5089 tablet_bootstrap.cc:492] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:28.364809 5092 tablet_bootstrap.cc:654] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.364817 5089 ts_tablet_manager.cc:1397] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.004s
I20250626 19:57:28.367120 5089 raft_consensus.cc:357] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:28.367792 5089 raft_consensus.cc:383] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.368225 5089 raft_consensus.cc:738] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.369062 4319 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_a6836de37d3b41c7aa82c0c1e14ab4b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:28.370723 4319 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6836de37d3b41c7aa82c0c1e14ab4b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:28.375216 4959 tablet_service.cc:1468] Processing CreateTablet for tablet af58f34e2248432fae5c084bd3388028 (DEFAULT_TABLE table=default.Table_e25c8c1da59041118a0072aa639e3771 [id=bea62f96f1cd4c2ab35704acf7def541]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:28.376214 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af58f34e2248432fae5c084bd3388028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:28.375897 5089 consensus_queue.cc:260] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:28.376688 4657 tablet_service.cc:1468] Processing CreateTablet for tablet 6fdf32e1756447c9a9e59956bb18b772 (DEFAULT_TABLE table=default.Table_689b779650c544c7baa707961e76e953 [id=49ced2ddeecf4d18a8693950b66b1e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:28.377720 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fdf32e1756447c9a9e59956bb18b772. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:28.380604 5089 ts_tablet_manager.cc:1428] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20250626 19:57:28.381449 5089 tablet_bootstrap.cc:492] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:28.388729 5148 raft_consensus.cc:491] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:28.389534 5089 tablet_bootstrap.cc:654] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.389200 5148 raft_consensus.cc:513] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.391847 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a8f494f8614e7ebb8ca880b6c80e78" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:28.393131 4912 leader_election.cc:343] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:28.394444 5148 leader_election.cc:290] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:28.399698 5188 consensus_queue.cc:1035] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:28.404076 4322 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_0b702c28aabb41ea8cd8b1c5e8ecef96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:28.403908 4321 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_4074ca47bf294632b7fe7f9958389a45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:28.406255 4322 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b702c28aabb41ea8cd8b1c5e8ecef96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:57:28.406425 4321 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4074ca47bf294632b7fe7f9958389a45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:28.410208 4956 tablet_service.cc:1468] Processing CreateTablet for tablet 6fdf32e1756447c9a9e59956bb18b772 (DEFAULT_TABLE table=default.Table_689b779650c544c7baa707961e76e953 [id=49ced2ddeecf4d18a8693950b66b1e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:28.411291 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fdf32e1756447c9a9e59956bb18b772. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:28.411875 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 44960f8db08942879cde6f96e9c620da (DEFAULT_TABLE table=default.Table_27bcd169607a45599dc5a644e6ea5dd7 [id=8df233b9523c49adb3d24727bda24b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:28.412892 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44960f8db08942879cde6f96e9c620da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:28.415845 4652 tablet_service.cc:1468] Processing CreateTablet for tablet 1e453a4f8ca44f4a940928d9f4fcd5be (DEFAULT_TABLE table=default.Table_f579d2c76b104fc88466bcdcd54d2603 [id=049afd5ef383451fa96431d71b4f2ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:28.416198 4655 tablet_service.cc:1468] Processing CreateTablet for tablet 7aee4118595346ef990a8a52eb007072 (DEFAULT_TABLE table=default.Table_443891de930144a484281a0abf7d60f2 [id=b5086ce274d44f4dae848455c6274ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:28.416893 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e453a4f8ca44f4a940928d9f4fcd5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:28.417416 4655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aee4118595346ef990a8a52eb007072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:28.421175 4954 tablet_service.cc:1468] Processing CreateTablet for tablet 44960f8db08942879cde6f96e9c620da (DEFAULT_TABLE table=default.Table_27bcd169607a45599dc5a644e6ea5dd7 [id=8df233b9523c49adb3d24727bda24b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:28.422338 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44960f8db08942879cde6f96e9c620da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:28.423844 4957 tablet_service.cc:1468] Processing CreateTablet for tablet 1e453a4f8ca44f4a940928d9f4fcd5be (DEFAULT_TABLE table=default.Table_f579d2c76b104fc88466bcdcd54d2603 [id=049afd5ef383451fa96431d71b4f2ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:28.425803 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e453a4f8ca44f4a940928d9f4fcd5be. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:28.435528 5064 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 19:57:24.707456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.46.129:39845 (local address 127.0.0.1:56460)
0626 19:57:24.735212 (+ 27756us) negotiation.cc:107] Waiting for socket to connect
0626 19:57:24.735269 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0626 19:57:24.738921 (+ 3652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0626 19:57:24.741000 (+ 2079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0626 19:57:24.741035 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0626 19:57:24.741171 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0626 19:57:24.742958 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0626 19:57:24.743011 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0626 19:57:24.746836 (+ 3825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0626 19:57:24.746879 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0626 19:57:28.425997 (+3679118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0626 19:57:28.426097 (+ 100us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0626 19:57:28.426507 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0626 19:57:28.431269 (+ 4762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0626 19:57:28.431296 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0626 19:57:28.431389 (+ 93us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0626 19:57:28.431533 (+ 144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0626 19:57:28.432287 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":26215,"thread_start_us":29424,"threads_started":1}
W20250626 19:57:28.439453 5065 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 19:57:24.707794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.4.46.129:39845)
0626 19:57:24.739455 (+ 31661us) server_negotiation.cc:206] Beginning negotiation
0626 19:57:24.739491 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0626 19:57:24.739565 (+ 74us) server_negotiation.cc:407] Connection header received
0626 19:57:24.740252 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0626 19:57:24.740295 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0626 19:57:24.740653 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0626 19:57:24.742189 (+ 1536us) server_negotiation.cc:226] Negotiated authn=SASL
0626 19:57:24.743299 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0626 19:57:24.745864 (+ 2565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0626 19:57:28.428262 (+3682398us) negotiation.cc:327] Negotiation complete: Timed out: Server connection negotiation failed: server connection from 127.0.0.1:56460: received 0 of 81 requested bytes
Metrics: {"server-negotiator.queue_time_us":30873,"thread_start_us":30513,"threads_started":1}
I20250626 19:57:28.456282 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 6fdf32e1756447c9a9e59956bb18b772 (DEFAULT_TABLE table=default.Table_689b779650c544c7baa707961e76e953 [id=49ced2ddeecf4d18a8693950b66b1e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:28.457407 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a8f494f8614e7ebb8ca880b6c80e78" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:28.458034 4821 raft_consensus.cc:2466] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:28.459271 4911 leader_election.cc:304] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:28.460482 4958 tablet_service.cc:1468] Processing CreateTablet for tablet 7aee4118595346ef990a8a52eb007072 (DEFAULT_TABLE table=default.Table_443891de930144a484281a0abf7d60f2 [id=b5086ce274d44f4dae848455c6274ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:28.461586 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aee4118595346ef990a8a52eb007072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:28.462424 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 1e453a4f8ca44f4a940928d9f4fcd5be (DEFAULT_TABLE table=default.Table_f579d2c76b104fc88466bcdcd54d2603 [id=049afd5ef383451fa96431d71b4f2ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:28.462916 5148 raft_consensus.cc:2802] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:28.463246 5148 raft_consensus.cc:491] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:28.463559 5148 raft_consensus.cc:3058] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:28.464181 4797 tablet_service.cc:1468] Processing CreateTablet for tablet 7aee4118595346ef990a8a52eb007072 (DEFAULT_TABLE table=default.Table_443891de930144a484281a0abf7d60f2 [id=b5086ce274d44f4dae848455c6274ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:24.748231 4282 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu with pid 4291
I20250626 19:57:28.467694 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 44960f8db08942879cde6f96e9c620da (DEFAULT_TABLE table=default.Table_27bcd169607a45599dc5a644e6ea5dd7 [id=8df233b9523c49adb3d24727bda24b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:28.476668 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fdf32e1756447c9a9e59956bb18b772. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:28.479346 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e453a4f8ca44f4a940928d9f4fcd5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:28.483274 4797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aee4118595346ef990a8a52eb007072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:28.485407 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44960f8db08942879cde6f96e9c620da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:28.485563 5148 raft_consensus.cc:513] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.487448 5148 leader_election.cc:290] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:28.491083 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a8f494f8614e7ebb8ca880b6c80e78" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:28.491645 4821 raft_consensus.cc:3058] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:28.499070 4821 raft_consensus.cc:2466] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:28.500331 4911 leader_election.cc:304] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:28.504313 5091 consensus_queue.cc:260] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.506830 5091 ts_tablet_manager.cc:1428] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.185s user 0.006s sys 0.001s
I20250626 19:57:28.514279 5089 tablet_bootstrap.cc:492] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:28.514676 5089 ts_tablet_manager.cc:1397] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.133s user 0.009s sys 0.011s
I20250626 19:57:28.515314 5148 raft_consensus.cc:2802] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:28.519842 5091 tablet_bootstrap.cc:492] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:28.523386 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a8f494f8614e7ebb8ca880b6c80e78" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:28.523442 5089 raft_consensus.cc:357] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.524053 5089 raft_consensus.cc:383] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.524307 5089 raft_consensus.cc:738] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.524884 5089 consensus_queue.cc:260] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:28.537528 4912 leader_election.cc:343] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:28.547595 5089 ts_tablet_manager.cc:1428] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20250626 19:57:28.548605 5089 tablet_bootstrap.cc:492] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:28.553277 5091 tablet_bootstrap.cc:654] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.554482 5089 tablet_bootstrap.cc:654] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.563656 5092 tablet_bootstrap.cc:492] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:28.564064 5092 ts_tablet_manager.cc:1397] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.335s user 0.013s sys 0.005s
I20250626 19:57:28.566229 5092 raft_consensus.cc:357] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.566902 5092 raft_consensus.cc:383] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.567157 5092 raft_consensus.cc:738] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.567924 5092 consensus_queue.cc:260] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.577872 5148 raft_consensus.cc:695] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:28.588613 5208 raft_consensus.cc:491] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:28.589102 5208 raft_consensus.cc:513] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.590914 5209 raft_consensus.cc:491] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:28.591543 5209 raft_consensus.cc:513] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.596383 5148 consensus_queue.cc:237] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.609061 5089 tablet_bootstrap.cc:492] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:28.608963 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bd8dae5699c4725915814044d76a257" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:28.609457 5089 ts_tablet_manager.cc:1397] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.061s user 0.012s sys 0.000s
I20250626 19:57:28.611726 5089 raft_consensus.cc:357] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.612344 5089 raft_consensus.cc:383] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.612586 5089 raft_consensus.cc:738] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.613155 5089 consensus_queue.cc:260] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.616209 5089 ts_tablet_manager.cc:1428] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250626 19:57:28.620301 5092 ts_tablet_manager.cc:1428] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.056s user 0.011s sys 0.002s
I20250626 19:57:28.621202 5092 tablet_bootstrap.cc:492] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:28.622172 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea7374ad6ba64e46b7df9852fa33f145" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:28.622735 4677 raft_consensus.cc:2466] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:28.623745 4912 leader_election.cc:304] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:28.624954 5211 raft_consensus.cc:2802] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:28.625136 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bd8dae5699c4725915814044d76a257" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:28.625877 4821 raft_consensus.cc:2391] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:28.624874 5208 leader_election.cc:290] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:28.627081 4911 leader_election.cc:304] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:28.627538 5209 leader_election.cc:290] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:28.627982 5209 raft_consensus.cc:3058] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:28.625293 5211 raft_consensus.cc:491] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:28.628942 5211 raft_consensus.cc:3058] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:28.628993 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea7374ad6ba64e46b7df9852fa33f145" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:28.629587 4821 raft_consensus.cc:2466] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:28.633237 5089 tablet_bootstrap.cc:492] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:28.637488 5092 tablet_bootstrap.cc:654] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.638815 5089 tablet_bootstrap.cc:654] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.639614 5091 tablet_bootstrap.cc:492] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:28.640132 5091 ts_tablet_manager.cc:1397] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.121s user 0.008s sys 0.006s
I20250626 19:57:28.642212 5091 raft_consensus.cc:357] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.645601 5091 raft_consensus.cc:383] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.645917 5091 raft_consensus.cc:738] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.637547 5209 raft_consensus.cc:2747] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:28.647712 5091 consensus_queue.cc:260] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.660617 5091 ts_tablet_manager.cc:1428] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20250626 19:57:28.668826 5211 raft_consensus.cc:513] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:28.670789 4870 heartbeater.cc:646] Failed to heartbeat to 127.4.46.190:33823 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.4.46.190:33823 timed out after 1.000s (SENT)
I20250626 19:57:28.671411 5211 leader_election.cc:290] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:28.672659 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea7374ad6ba64e46b7df9852fa33f145" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:28.673194 4677 raft_consensus.cc:3058] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:28.676990 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea7374ad6ba64e46b7df9852fa33f145" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:28.677532 4821 raft_consensus.cc:3058] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:28.684223 4821 raft_consensus.cc:2466] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:28.685374 4911 leader_election.cc:304] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:28.690325 4677 raft_consensus.cc:2466] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:28.692405 5188 raft_consensus.cc:491] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:28.692842 5188 raft_consensus.cc:513] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:28.694650 5188 leader_election.cc:290] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:28.695518 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "499f6ae3262940b4859c14a53722482f" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:28.696241 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "499f6ae3262940b4859c14a53722482f" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:28.696628 5092 tablet_bootstrap.cc:492] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
W20250626 19:57:28.696673 4755 leader_election.cc:343] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:28.696995 5092 ts_tablet_manager.cc:1397] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.002s
W20250626 19:57:28.697621 4756 leader_election.cc:343] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:28.698031 4756 leader_election.cc:304] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:28.699098 5188 raft_consensus.cc:2747] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:28.699307 5092 raft_consensus.cc:357] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:28.699615 5211 raft_consensus.cc:2802] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:28.699920 5092 raft_consensus.cc:383] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.700179 5092 raft_consensus.cc:738] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.711421 5092 consensus_queue.cc:260] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:28.713508 5092 ts_tablet_manager.cc:1428] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250626 19:57:28.714411 5092 tablet_bootstrap.cc:492] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:28.700490 5211 raft_consensus.cc:695] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:28.716449 5211 consensus_queue.cc:237] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.717367 5091 tablet_bootstrap.cc:492] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:28.722626 5089 tablet_bootstrap.cc:492] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:28.725823 5089 ts_tablet_manager.cc:1397] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.093s user 0.034s sys 0.039s
I20250626 19:57:28.729702 5089 raft_consensus.cc:357] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.733249 5091 tablet_bootstrap.cc:654] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.733580 5089 raft_consensus.cc:383] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250626 19:57:28.734262 4912 connection.cc:537] server connection from 127.0.0.1:51044 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250626 19:57:28.744338 5092 tablet_bootstrap.cc:654] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.744771 5089 raft_consensus.cc:738] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.745569 5168 consensus_queue.cc:1035] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:28.745532 5089 consensus_queue.cc:260] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.748797 5089 ts_tablet_manager.cc:1428] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20250626 19:57:28.749713 5089 tablet_bootstrap.cc:492] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:28.755285 5089 tablet_bootstrap.cc:654] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:28.756796 4611 connection.cc:537] server connection from 127.0.0.1:56472 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 19:57:28.759581 4727 heartbeater.cc:646] Failed to heartbeat to 127.4.46.190:33823 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.4.46.190:33823 (error 108)
W20250626 19:57:28.756137 4755 connection.cc:537] server connection from 127.0.0.1:36858 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 19:57:28.760349 5034 connection.cc:537] client connection to 127.4.46.190:33823 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 19:57:28.762384 5034 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.46.190:33823)
W20250626 19:57:28.763286 5026 heartbeater.cc:646] Failed to heartbeat to 127.4.46.190:33823 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.4.46.190:33823 (error 108)
W20250626 19:57:28.764878 4754 connection.cc:537] client connection to 127.4.46.190:33823 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 19:57:28.767675 4464 connection.cc:537] server connection from 127.0.0.1:56326 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 19:57:28.769798 5034 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.46.190:33823)
W20250626 19:57:28.771104 5034 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.46.190:33823)
W20250626 19:57:28.774374 5034 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.46.190:33823)
I20250626 19:57:28.778527 5092 tablet_bootstrap.cc:492] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:28.778996 5092 ts_tablet_manager.cc:1397] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.065s user 0.016s sys 0.000s
I20250626 19:57:28.781512 5092 raft_consensus.cc:357] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:28.782254 5092 raft_consensus.cc:383] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.782539 5092 raft_consensus.cc:738] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.783187 5092 consensus_queue.cc:260] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
W20250626 19:57:28.785454 4384 connection.cc:537] server connection from 127.0.0.1:35872 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250626 19:57:28.785818 5092 ts_tablet_manager.cc:1428] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250626 19:57:28.787976 5092 tablet_bootstrap.cc:492] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:28.793841 5092 tablet_bootstrap.cc:654] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:28.795681 5221 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 19:57:28.764644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.46.190:33823 (local address 127.4.46.131:56617)
0626 19:57:28.793453 (+ 28809us) negotiation.cc:107] Waiting for socket to connect
0626 19:57:28.793741 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":28123,"thread_start_us":29538,"threads_started":1}
I20250626 19:57:28.800369 5091 tablet_bootstrap.cc:492] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:28.800819 5091 ts_tablet_manager.cc:1397] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.086s user 0.008s sys 0.019s
I20250626 19:57:28.803495 5091 raft_consensus.cc:357] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:28.804222 5223 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 19:57:28.768783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.46.190:33823 (local address 127.4.46.130:33989)
0626 19:57:28.802638 (+ 33855us) negotiation.cc:107] Waiting for socket to connect
0626 19:57:28.802987 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":33067,"threads_started":1}
I20250626 19:57:28.804350 5091 raft_consensus.cc:383] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.804672 5091 raft_consensus.cc:738] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.805532 5091 consensus_queue.cc:260] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.821256 5091 ts_tablet_manager.cc:1428] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.020s user 0.003s sys 0.004s
I20250626 19:57:28.822167 5091 tablet_bootstrap.cc:492] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:28.838120 5091 tablet_bootstrap.cc:654] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.855633 5091 tablet_bootstrap.cc:492] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:28.857143 5091 ts_tablet_manager.cc:1397] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.004s
I20250626 19:57:28.859608 5091 raft_consensus.cc:357] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:28.860211 5091 raft_consensus.cc:383] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.860452 5091 raft_consensus.cc:738] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.860965 5091 consensus_queue.cc:260] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:28.866014 5091 ts_tablet_manager.cc:1428] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250626 19:57:28.866863 5091 tablet_bootstrap.cc:492] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
W20250626 19:57:28.879823 5219 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 19:57:28.761553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.46.190:33823 (local address 127.4.46.129:48111)
0626 19:57:28.878306 (+116753us) negotiation.cc:107] Waiting for socket to connect
0626 19:57:28.878657 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":115858,"thread_start_us":115431,"threads_started":1}
I20250626 19:57:28.880671 5176 consensus_queue.cc:1035] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:57:28.882687 5224 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 19:57:28.785887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.46.190:33823 (local address 127.0.0.1:35048)
0626 19:57:28.880911 (+ 95024us) negotiation.cc:107] Waiting for socket to connect
0626 19:57:28.881403 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":19680,"thread_start_us":21752,"threads_started":1}
I20250626 19:57:28.819792 5089 tablet_bootstrap.cc:492] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:28.887178 5089 ts_tablet_manager.cc:1397] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.138s user 0.018s sys 0.004s
I20250626 19:57:28.878680 5091 tablet_bootstrap.cc:654] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.889240 5176 consensus_queue.cc:1035] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:28.890142 5089 raft_consensus.cc:357] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.891541 5089 raft_consensus.cc:383] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.891989 5089 raft_consensus.cc:738] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.892963 5089 consensus_queue.cc:260] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.897980 5089 ts_tablet_manager.cc:1428] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250626 19:57:28.899250 5089 tablet_bootstrap.cc:492] T 44960f8db08942879cde6f96e9c620da P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:28.929212 5091 tablet_bootstrap.cc:492] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:28.929641 5091 ts_tablet_manager.cc:1397] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.063s user 0.007s sys 0.010s
I20250626 19:57:28.931974 5091 raft_consensus.cc:357] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:28.932622 5091 raft_consensus.cc:383] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.932884 5091 raft_consensus.cc:738] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.935019 5091 consensus_queue.cc:260] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:28.940507 5091 ts_tablet_manager.cc:1428] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250626 19:57:28.943902 5091 tablet_bootstrap.cc:492] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:28.952194 5091 tablet_bootstrap.cc:654] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:28.966298 5091 tablet_bootstrap.cc:492] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:28.966281 5230 raft_consensus.cc:491] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:28.966706 5091 ts_tablet_manager.cc:1397] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.010s
I20250626 19:57:28.966784 5230 raft_consensus.cc:513] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:28.968972 5091 raft_consensus.cc:357] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.969617 5091 raft_consensus.cc:383] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.969877 5091 raft_consensus.cc:738] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.970855 5233 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:28.971303 5233 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:28.970484 5091 consensus_queue.cc:260] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.973217 5233 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
W20250626 19:57:28.978324 4462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection refused (error 111)
I20250626 19:57:28.981144 5092 tablet_bootstrap.cc:492] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:28.981546 5092 ts_tablet_manager.cc:1397] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.194s user 0.069s sys 0.086s
I20250626 19:57:28.982158 5168 raft_consensus.cc:491] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:28.982765 5168 raft_consensus.cc:513] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.983289 5176 consensus_queue.cc:1035] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:28.984735 5168 leader_election.cc:290] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:28.986196 5092 raft_consensus.cc:357] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.987090 5092 raft_consensus.cc:383] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:28.987413 5092 raft_consensus.cc:738] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:28.987706 5153 raft_consensus.cc:491] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:28.988150 5153 raft_consensus.cc:513] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:28.988425 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a8f494f8614e7ebb8ca880b6c80e78" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:28.988139 5092 consensus_queue.cc:260] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:28.990037 5153 leader_election.cc:290] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:28.992151 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afd1fafb53434957b002fad5c77431d4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:28.992730 4677 raft_consensus.cc:2466] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:28.994457 4755 leader_election.cc:304] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:28.995462 5188 raft_consensus.cc:2802] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:28.995774 5188 raft_consensus.cc:491] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:28.996088 5188 raft_consensus.cc:3058] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:28.996146 5091 ts_tablet_manager.cc:1428] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.029s user 0.004s sys 0.000s
I20250626 19:57:28.999050 5091 tablet_bootstrap.cc:492] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:29.000802 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afd1fafb53434957b002fad5c77431d4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:29.001415 4977 raft_consensus.cc:2466] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:29.004675 5188 raft_consensus.cc:513] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.007383 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afd1fafb53434957b002fad5c77431d4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:29.007938 4677 raft_consensus.cc:3058] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.009092 4407 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 6 candidate_status { last_received { term: 5 index: 84 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:29.009931 4407 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae949d1fcdf54e87930871ac0ae7da31 for term 6 because replica has last-logged OpId of term: 5 index: 85, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 84.
I20250626 19:57:29.010888 5091 tablet_bootstrap.cc:654] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.013579 5230 leader_election.cc:290] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
W20250626 19:57:29.032984 4462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.190:33823, user_credentials={real_user=slave}} blocked reactor thread for 54807.7us
I20250626 19:57:29.036583 5089 tablet_bootstrap.cc:654] T 44960f8db08942879cde6f96e9c620da P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.030618 4677 raft_consensus.cc:2466] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:29.040014 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afd1fafb53434957b002fad5c77431d4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:29.040596 4977 raft_consensus.cc:3058] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.040681 4755 leader_election.cc:304] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:29.041608 5153 raft_consensus.cc:2802] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:29.042310 5153 raft_consensus.cc:695] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:29.043109 5153 consensus_queue.cc:237] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.049904 4977 raft_consensus.cc:2466] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:29.053174 5160 raft_consensus.cc:491] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:29.053638 5160 raft_consensus.cc:513] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.055312 5092 ts_tablet_manager.cc:1428] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.073s user 0.008s sys 0.000s
I20250626 19:57:29.055470 5160 leader_election.cc:290] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:29.058315 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cb6bda529624ac5b5a87a03b4a02d56" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:29.059681 4462 leader_election.cc:336] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823): Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection refused (error 111)
W20250626 19:57:29.059875 4756 leader_election.cc:343] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:29.060201 4462 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31; no voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:29.061410 5233 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 19:57:29.061637 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cb6bda529624ac5b5a87a03b4a02d56" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:29.062510 5188 leader_election.cc:290] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
W20250626 19:57:29.063009 4755 leader_election.cc:343] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:29.063575 4755 leader_election.cc:304] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:29.064929 5160 raft_consensus.cc:2747] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:29.056478 5092 tablet_bootstrap.cc:492] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
W20250626 19:57:29.076630 4383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20250626 19:57:29.077636 4487 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ac80df82774b9b87c0be076b5637af" candidate_term: 6 candidate_status { last_received { term: 5 index: 85 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31" is_pre_election: true
I20250626 19:57:29.078202 4487 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62ac80df82774b9b87c0be076b5637af in term 5.
W20250626 19:57:29.081100 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 99295.7us
I20250626 19:57:29.083874 5089 tablet_bootstrap.cc:492] T 44960f8db08942879cde6f96e9c620da P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:29.084271 5089 ts_tablet_manager.cc:1397] T 44960f8db08942879cde6f96e9c620da P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.185s user 0.011s sys 0.000s
I20250626 19:57:29.086457 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a8f494f8614e7ebb8ca880b6c80e78" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:29.087260 4821 raft_consensus.cc:2391] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:29.087215 5089 raft_consensus.cc:357] T 44960f8db08942879cde6f96e9c620da P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.087960 5089 raft_consensus.cc:383] T 44960f8db08942879cde6f96e9c620da P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.088222 5089 raft_consensus.cc:738] T 44960f8db08942879cde6f96e9c620da P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.089812 4383 leader_election.cc:304] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ac80df82774b9b87c0be076b5637af, ae949d1fcdf54e87930871ac0ae7da31; no voters:
I20250626 19:57:29.090979 5230 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 19:57:29.091292 5230 raft_consensus.cc:491] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 5 FOLLOWER]: Starting leader election (detected failure of leader e6c58006b6be41c1bd6bdb5eb010d360)
I20250626 19:57:29.091636 5230 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 5 FOLLOWER]: Advancing to term 6
I20250626 19:57:29.096510 4609 leader_election.cc:304] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:29.098611 5176 raft_consensus.cc:3058] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.102052 5230 raft_consensus.cc:513] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:29.105228 5176 raft_consensus.cc:2747] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:29.093513 4383 leader_election.cc:336] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823): Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection refused (error 111)
I20250626 19:57:29.108124 5091 tablet_bootstrap.cc:492] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:29.111774 5091 ts_tablet_manager.cc:1397] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.113s user 0.012s sys 0.027s
I20250626 19:57:29.114876 5092 tablet_bootstrap.cc:654] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.114501 5091 raft_consensus.cc:357] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.115217 5176 raft_consensus.cc:491] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:29.115773 5091 raft_consensus.cc:383] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.116032 5091 raft_consensus.cc:738] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.116124 5176 raft_consensus.cc:513] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.116679 5091 consensus_queue.cc:260] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.117542 5089 consensus_queue.cc:260] T 44960f8db08942879cde6f96e9c620da P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.120163 5089 ts_tablet_manager.cc:1428] T 44960f8db08942879cde6f96e9c620da P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.036s user 0.005s sys 0.002s
I20250626 19:57:29.120915 5089 tablet_bootstrap.cc:492] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:29.124390 5153 raft_consensus.cc:491] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:29.124805 5153 raft_consensus.cc:513] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.126955 5153 leader_election.cc:290] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:29.127422 5091 ts_tablet_manager.cc:1428] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20250626 19:57:29.128623 5230 leader_election.cc:290] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 6 election: Requested vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:29.128913 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f91a78f6d474133976242ea8d259e29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:29.130967 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512dc911bfce49b29c6fc8cd90523723" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:29.131088 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f91a78f6d474133976242ea8d259e29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:29.131564 4978 raft_consensus.cc:2466] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:29.131664 4977 raft_consensus.cc:2466] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:29.132771 4487 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ac80df82774b9b87c0be076b5637af" candidate_term: 6 candidate_status { last_received { term: 5 index: 85 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31"
I20250626 19:57:29.133304 4487 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 5 FOLLOWER]: Advancing to term 6
I20250626 19:57:29.133730 5091 tablet_bootstrap.cc:492] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:29.133607 4756 leader_election.cc:304] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
W20250626 19:57:29.134143 4383 leader_election.cc:336] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823): Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection refused (error 111)
I20250626 19:57:29.134658 4611 leader_election.cc:304] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:29.135396 5236 raft_consensus.cc:2802] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:29.135720 5236 raft_consensus.cc:491] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:29.136024 5236 raft_consensus.cc:3058] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:29.139056 4755 leader_election.cc:343] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:29.139082 5153 raft_consensus.cc:2802] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:29.139582 5153 raft_consensus.cc:491] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:29.139930 5153 raft_consensus.cc:3058] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.145823 4487 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62ac80df82774b9b87c0be076b5637af in term 6.
I20250626 19:57:29.152405 5153 raft_consensus.cc:513] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.153681 5188 raft_consensus.cc:491] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:29.154191 5188 raft_consensus.cc:513] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.155328 5153 leader_election.cc:290] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:29.156011 5188 leader_election.cc:290] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:29.156772 5089 tablet_bootstrap.cc:654] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.157089 4383 leader_election.cc:304] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62ac80df82774b9b87c0be076b5637af, ae949d1fcdf54e87930871ac0ae7da31; no voters: e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:29.157372 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f91a78f6d474133976242ea8d259e29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:29.157979 4977 raft_consensus.cc:3058] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.158330 5230 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 6 FOLLOWER]: Leader election won for term 6
I20250626 19:57:29.159967 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b3bed19e7e34d24b995840a3bde5ef0" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:29.160321 5092 tablet_bootstrap.cc:492] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:29.160647 5092 ts_tablet_manager.cc:1397] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.104s user 0.008s sys 0.000s
W20250626 19:57:29.165588 4756 leader_election.cc:343] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:29.167599 5092 raft_consensus.cc:357] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.168219 5092 raft_consensus.cc:383] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.168385 5153 raft_consensus.cc:491] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:29.168464 5092 raft_consensus.cc:738] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.168812 5153 raft_consensus.cc:513] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.169409 5176 leader_election.cc:290] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:29.171655 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512dc911bfce49b29c6fc8cd90523723" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:29.172212 4821 raft_consensus.cc:2466] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:29.172662 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512dc911bfce49b29c6fc8cd90523723" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:29.173203 4978 raft_consensus.cc:2466] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:29.173386 5091 tablet_bootstrap.cc:654] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.176246 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b3bed19e7e34d24b995840a3bde5ef0" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:29.176831 4677 raft_consensus.cc:2466] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:29.179207 4756 leader_election.cc:304] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:29.180069 5160 raft_consensus.cc:2802] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:29.179955 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512dc911bfce49b29c6fc8cd90523723" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:29.180384 5160 raft_consensus.cc:491] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:29.180708 5160 raft_consensus.cc:3058] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.184223 5236 raft_consensus.cc:513] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.185381 4674 raft_consensus.cc:2391] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93f885c6802341c889a40e9c20fbfff1 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:29.188413 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f91a78f6d474133976242ea8d259e29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:29.188288 5160 raft_consensus.cc:513] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.170629 5153 leader_election.cc:290] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:29.190640 4755 leader_election.cc:304] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:29.192060 5236 leader_election.cc:290] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:29.192458 4755 leader_election.cc:343] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:29.193116 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512dc911bfce49b29c6fc8cd90523723" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:29.193464 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512dc911bfce49b29c6fc8cd90523723" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:29.194026 4978 raft_consensus.cc:3058] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.200191 5092 consensus_queue.cc:260] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.205075 4977 raft_consensus.cc:2466] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:29.205708 5188 raft_consensus.cc:2802] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:29.206143 5188 raft_consensus.cc:491] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:29.206513 5188 raft_consensus.cc:3058] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.222239 4821 raft_consensus.cc:2391] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 93f885c6802341c889a40e9c20fbfff1 in this term.
I20250626 19:57:29.224423 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512dc911bfce49b29c6fc8cd90523723" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:29.225188 4676 raft_consensus.cc:2391] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 93f885c6802341c889a40e9c20fbfff1 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:29.225641 4756 leader_election.cc:304] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:29.226552 5160 leader_election.cc:290] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:29.227171 5160 raft_consensus.cc:2802] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:29.227797 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512dc911bfce49b29c6fc8cd90523723" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:29.229146 5160 raft_consensus.cc:695] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:29.229413 4756 leader_election.cc:304] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:29.230245 5160 consensus_queue.cc:237] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.231489 5153 raft_consensus.cc:2747] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:57:29.235682 5091 tablet_bootstrap.cc:492] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:29.235658 5230 raft_consensus.cc:695] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [term 6 LEADER]: Becoming Leader. State: Replica: 62ac80df82774b9b87c0be076b5637af, State: Running, Role: LEADER
I20250626 19:57:29.235585 4978 raft_consensus.cc:2466] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:29.236075 5091 ts_tablet_manager.cc:1397] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.103s user 0.007s sys 0.008s
I20250626 19:57:29.236668 5230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 5.85, Last appended by leader: 85, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:29.237162 4611 leader_election.cc:304] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters: 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:29.238554 5236 raft_consensus.cc:2802] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:29.238246 5091 raft_consensus.cc:357] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.238961 5091 raft_consensus.cc:383] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.239218 5091 raft_consensus.cc:738] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.240113 5236 raft_consensus.cc:695] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:29.209088 5092 ts_tablet_manager.cc:1428] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.048s user 0.001s sys 0.005s
I20250626 19:57:29.244271 5092 tablet_bootstrap.cc:492] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:29.243993 5091 consensus_queue.cc:260] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.238332 5188 raft_consensus.cc:513] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.246865 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b3bed19e7e34d24b995840a3bde5ef0" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:29.247402 4676 raft_consensus.cc:3058] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.254112 4676 raft_consensus.cc:2466] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:29.254736 5089 tablet_bootstrap.cc:492] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:29.255153 5089 ts_tablet_manager.cc:1397] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.134s user 0.010s sys 0.000s
I20250626 19:57:29.255344 4755 leader_election.cc:304] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:29.256332 5247 raft_consensus.cc:2802] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:29.259747 5091 ts_tablet_manager.cc:1428] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.023s user 0.002s sys 0.004s
I20250626 19:57:29.257274 5089 raft_consensus.cc:357] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.263566 5089 raft_consensus.cc:383] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.263207 5247 raft_consensus.cc:695] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:29.265442 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b3bed19e7e34d24b995840a3bde5ef0" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:29.266044 4978 raft_consensus.cc:3058] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.266389 5091 tablet_bootstrap.cc:492] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:29.269059 5236 consensus_queue.cc:237] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.263828 5089 raft_consensus.cc:738] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.271230 5089 consensus_queue.cc:260] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.271430 5247 consensus_queue.cc:237] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.273360 4978 raft_consensus.cc:2466] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:29.276449 5188 leader_election.cc:290] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:29.283602 5153 raft_consensus.cc:491] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:29.283836 5092 tablet_bootstrap.cc:654] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.284056 5153 raft_consensus.cc:513] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.282466 5244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ac80df82774b9b87c0be076b5637af. Latest consensus state: current_term: 6 leader_uuid: "62ac80df82774b9b87c0be076b5637af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:29.285532 5244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: This master's current role is: LEADER
I20250626 19:57:29.285970 5153 leader_election.cc:290] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:29.287945 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fdf32e1756447c9a9e59956bb18b772" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:29.290987 4755 leader_election.cc:343] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:29.291872 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fdf32e1756447c9a9e59956bb18b772" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:29.293053 4756 leader_election.cc:343] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:29.293427 4756 leader_election.cc:304] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:29.294528 5153 raft_consensus.cc:2747] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:29.294840 5089 ts_tablet_manager.cc:1428] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20250626 19:57:29.296049 5089 tablet_bootstrap.cc:492] T 7aee4118595346ef990a8a52eb007072 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:29.318897 5091 tablet_bootstrap.cc:654] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.321485 5211 raft_consensus.cc:491] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:29.322070 5211 raft_consensus.cc:513] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.324229 5211 leader_election.cc:290] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:29.326105 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "499f6ae3262940b4859c14a53722482f" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:29.326696 4676 raft_consensus.cc:2466] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:29.327587 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "499f6ae3262940b4859c14a53722482f" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:29.327780 4912 leader_election.cc:304] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:29.328146 4821 raft_consensus.cc:2466] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:29.328544 5211 raft_consensus.cc:2802] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:29.328864 5211 raft_consensus.cc:491] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:29.329174 5211 raft_consensus.cc:3058] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.333576 5089 tablet_bootstrap.cc:654] T 7aee4118595346ef990a8a52eb007072 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.347100 5236 raft_consensus.cc:491] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:29.347575 5236 raft_consensus.cc:513] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.347885 5211 raft_consensus.cc:513] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.349342 5236 leader_election.cc:290] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:29.351831 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "499f6ae3262940b4859c14a53722482f" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:29.352221 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "499f6ae3262940b4859c14a53722482f" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:29.352382 4676 raft_consensus.cc:3058] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.352775 4820 raft_consensus.cc:3058] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.355109 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d8ffa9b658c413fa06d02ee6635dd1b" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:29.355684 4821 raft_consensus.cc:2466] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:29.357607 5091 tablet_bootstrap.cc:492] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:29.357683 4609 leader_election.cc:304] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:29.358024 5091 ts_tablet_manager.cc:1397] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.092s user 0.005s sys 0.003s
I20250626 19:57:29.358764 5236 raft_consensus.cc:2802] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:29.359099 5236 raft_consensus.cc:491] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:29.359442 5236 raft_consensus.cc:3058] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.359587 4820 raft_consensus.cc:2466] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:29.360710 4911 leader_election.cc:304] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:29.363801 5089 tablet_bootstrap.cc:492] T 7aee4118595346ef990a8a52eb007072 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:29.364207 5089 ts_tablet_manager.cc:1397] T 7aee4118595346ef990a8a52eb007072 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.000s
I20250626 19:57:29.364671 5091 raft_consensus.cc:357] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.365363 5091 raft_consensus.cc:383] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.365636 5091 raft_consensus.cc:738] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.365996 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d8ffa9b658c413fa06d02ee6635dd1b" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:29.366206 5091 consensus_queue.cc:260] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.366603 5089 raft_consensus.cc:357] T 7aee4118595346ef990a8a52eb007072 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.367241 5089 raft_consensus.cc:383] T 7aee4118595346ef990a8a52eb007072 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.367491 5089 raft_consensus.cc:738] T 7aee4118595346ef990a8a52eb007072 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.368052 5089 consensus_queue.cc:260] T 7aee4118595346ef990a8a52eb007072 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.370148 5089 ts_tablet_manager.cc:1428] T 7aee4118595346ef990a8a52eb007072 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20250626 19:57:29.370234 5250 log.cc:927] Time spent T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20250626 19:57:29.371017 5089 tablet_bootstrap.cc:492] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:29.371919 5211 leader_election.cc:290] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:29.372449 5211 raft_consensus.cc:2802] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:29.372877 5211 raft_consensus.cc:695] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:29.373623 5211 consensus_queue.cc:237] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.379202 5091 ts_tablet_manager.cc:1428] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20250626 19:57:29.380359 5091 tablet_bootstrap.cc:492] T 7aee4118595346ef990a8a52eb007072 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
W20250626 19:57:29.381381 4611 leader_election.cc:343] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:29.387704 4676 raft_consensus.cc:2466] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:29.389698 5236 raft_consensus.cc:513] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.392381 5089 tablet_bootstrap.cc:654] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.392411 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d8ffa9b658c413fa06d02ee6635dd1b" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
W20250626 19:57:29.393927 4611 leader_election.cc:343] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:29.394217 5236 leader_election.cc:290] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:29.398130 5092 tablet_bootstrap.cc:492] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:29.398554 5092 ts_tablet_manager.cc:1397] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.154s user 0.010s sys 0.004s
I20250626 19:57:29.399403 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d8ffa9b658c413fa06d02ee6635dd1b" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:29.400048 4820 raft_consensus.cc:3058] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.401055 5092 raft_consensus.cc:357] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.401687 5092 raft_consensus.cc:383] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.401943 5092 raft_consensus.cc:738] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.402547 5092 consensus_queue.cc:260] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.408504 5091 tablet_bootstrap.cc:654] T 7aee4118595346ef990a8a52eb007072 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.410773 4820 raft_consensus.cc:2466] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:29.412604 4609 leader_election.cc:304] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:29.420853 5089 tablet_bootstrap.cc:492] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:29.421289 5089 ts_tablet_manager.cc:1397] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.050s user 0.015s sys 0.000s
I20250626 19:57:29.421499 5092 ts_tablet_manager.cc:1428] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.023s user 0.004s sys 0.001s
I20250626 19:57:29.422298 5092 tablet_bootstrap.cc:492] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:29.423535 5089 raft_consensus.cc:357] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.424170 5089 raft_consensus.cc:383] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.424403 5089 raft_consensus.cc:738] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.424935 5089 consensus_queue.cc:260] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.426853 5089 ts_tablet_manager.cc:1428] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250626 19:57:29.427641 5089 tablet_bootstrap.cc:492] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:29.428195 5092 tablet_bootstrap.cc:654] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.433421 5089 tablet_bootstrap.cc:654] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.441911 5092 tablet_bootstrap.cc:492] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:29.442312 5092 ts_tablet_manager.cc:1397] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.000s
I20250626 19:57:29.444512 5092 raft_consensus.cc:357] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.445132 5092 raft_consensus.cc:383] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.445394 5092 raft_consensus.cc:738] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.446249 5236 raft_consensus.cc:2802] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:29.449963 5092 consensus_queue.cc:260] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.453480 5092 ts_tablet_manager.cc:1428] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.011s user 0.000s sys 0.005s
I20250626 19:57:29.454309 5092 tablet_bootstrap.cc:492] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:29.454236 5091 tablet_bootstrap.cc:492] T 7aee4118595346ef990a8a52eb007072 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:29.454782 5091 ts_tablet_manager.cc:1397] T 7aee4118595346ef990a8a52eb007072 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.075s user 0.009s sys 0.002s
I20250626 19:57:29.455366 5236 raft_consensus.cc:695] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:29.457396 5236 consensus_queue.cc:237] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.467240 5092 tablet_bootstrap.cc:654] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.457845 5091 raft_consensus.cc:357] T 7aee4118595346ef990a8a52eb007072 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.476171 5091 raft_consensus.cc:383] T 7aee4118595346ef990a8a52eb007072 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.476511 5089 tablet_bootstrap.cc:492] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:29.476464 5091 raft_consensus.cc:738] T 7aee4118595346ef990a8a52eb007072 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.476958 5089 ts_tablet_manager.cc:1397] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.011s
I20250626 19:57:29.479285 5089 raft_consensus.cc:357] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.479403 5091 consensus_queue.cc:260] T 7aee4118595346ef990a8a52eb007072 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.481367 5089 raft_consensus.cc:383] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.481701 5089 raft_consensus.cc:738] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.482189 5091 ts_tablet_manager.cc:1428] T 7aee4118595346ef990a8a52eb007072 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.027s user 0.004s sys 0.001s
I20250626 19:57:29.482461 5089 consensus_queue.cc:260] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.489177 5091 tablet_bootstrap.cc:492] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:29.489177 5089 ts_tablet_manager.cc:1428] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20250626 19:57:29.507151 5091 tablet_bootstrap.cc:654] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.531828 5092 tablet_bootstrap.cc:492] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:29.532219 5092 ts_tablet_manager.cc:1397] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.078s user 0.008s sys 0.000s
I20250626 19:57:29.534413 5092 raft_consensus.cc:357] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.535137 5092 raft_consensus.cc:383] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.535398 5092 raft_consensus.cc:738] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.536031 5092 consensus_queue.cc:260] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.538554 5092 ts_tablet_manager.cc:1428] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20250626 19:57:29.540012 5092 tablet_bootstrap.cc:492] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:29.546669 5092 tablet_bootstrap.cc:654] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.566067 5092 tablet_bootstrap.cc:492] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:29.566578 5092 ts_tablet_manager.cc:1397] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.003s
I20250626 19:57:29.568838 5092 raft_consensus.cc:357] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.581518 5092 raft_consensus.cc:383] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.581949 5092 raft_consensus.cc:738] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.582683 5092 consensus_queue.cc:260] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.584468 5091 tablet_bootstrap.cc:492] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:29.584915 5091 ts_tablet_manager.cc:1397] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.102s user 0.008s sys 0.001s
I20250626 19:57:29.587280 5091 raft_consensus.cc:357] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.587914 5091 raft_consensus.cc:383] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.588191 5092 ts_tablet_manager.cc:1428] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.021s user 0.003s sys 0.003s
I20250626 19:57:29.588178 5091 raft_consensus.cc:738] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.589586 4282 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
/tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.190:33823
--webserver_interface=127.4.46.190
--webserver_port=36587
--builtin_ntp_servers=127.4.46.148:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.46.190:33823,127.4.46.189:45039,127.4.46.188:40801
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250626 19:57:29.591017 5091 consensus_queue.cc:260] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.593526 5091 ts_tablet_manager.cc:1428] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250626 19:57:29.609061 5092 tablet_bootstrap.cc:492] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:29.613109 5091 tablet_bootstrap.cc:492] T 44960f8db08942879cde6f96e9c620da P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:29.614344 5092 tablet_bootstrap.cc:654] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.619578 5211 consensus_queue.cc:1035] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:29.624641 5211 consensus_queue.cc:1035] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:29.647001 5236 raft_consensus.cc:491] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:29.647491 5236 raft_consensus.cc:513] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.650462 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af58f34e2248432fae5c084bd3388028" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:29.651079 4821 raft_consensus.cc:2466] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:29.651117 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af58f34e2248432fae5c084bd3388028" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:29.652227 4609 leader_election.cc:304] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:29.652701 5236 leader_election.cc:290] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
W20250626 19:57:29.652788 4611 leader_election.cc:343] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:29.653519 5236 raft_consensus.cc:2802] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:29.653843 5236 raft_consensus.cc:491] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:29.654153 5236 raft_consensus.cc:3058] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.664964 5091 tablet_bootstrap.cc:654] T 44960f8db08942879cde6f96e9c620da P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.670158 4487 raft_consensus.cc:1273] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Refusing update from remote peer 62ac80df82774b9b87c0be076b5637af: Log matching property violated. Preceding OpId in replica: term: 5 index: 84. Preceding OpId from leader: term: 6 index: 86. (index mismatch)
I20250626 19:57:29.671770 5244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 84, Time since last communication: 0.001s
I20250626 19:57:29.675906 5236 raft_consensus.cc:513] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.683763 5092 tablet_bootstrap.cc:492] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:29.685231 5092 ts_tablet_manager.cc:1397] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.076s user 0.014s sys 0.017s
I20250626 19:57:29.686188 5211 raft_consensus.cc:491] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:29.686640 5211 raft_consensus.cc:513] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.689380 5091 tablet_bootstrap.cc:492] T 44960f8db08942879cde6f96e9c620da P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:29.689719 5211 leader_election.cc:290] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:29.689841 5091 ts_tablet_manager.cc:1397] T 44960f8db08942879cde6f96e9c620da P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.000s
I20250626 19:57:29.691143 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44960f8db08942879cde6f96e9c620da" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:29.692350 4912 leader_election.cc:343] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:29.689129 5092 raft_consensus.cc:357] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.692457 5091 raft_consensus.cc:357] T 44960f8db08942879cde6f96e9c620da P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.693076 5091 raft_consensus.cc:383] T 44960f8db08942879cde6f96e9c620da P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.693398 5091 raft_consensus.cc:738] T 44960f8db08942879cde6f96e9c620da P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.693346 5092 raft_consensus.cc:383] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.695325 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44960f8db08942879cde6f96e9c620da" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:29.696038 4820 raft_consensus.cc:2466] T 44960f8db08942879cde6f96e9c620da P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:29.698922 5092 raft_consensus.cc:738] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.701176 4911 leader_election.cc:304] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:29.701282 5255 consensus_queue.cc:1035] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:29.702688 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af58f34e2248432fae5c084bd3388028" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:29.704339 5236 leader_election.cc:290] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:29.707938 5091 consensus_queue.cc:260] T 44960f8db08942879cde6f96e9c620da P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.709684 5092 consensus_queue.cc:260] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.710565 4978 raft_consensus.cc:3058] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.708402 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af58f34e2248432fae5c084bd3388028" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:29.712512 4820 raft_consensus.cc:3058] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.717763 4978 raft_consensus.cc:2466] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:29.721416 4611 leader_election.cc:304] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:29.721920 5091 ts_tablet_manager.cc:1428] T 44960f8db08942879cde6f96e9c620da P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.032s user 0.006s sys 0.001s
I20250626 19:57:29.722244 5236 raft_consensus.cc:2802] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:29.722819 5091 tablet_bootstrap.cc:492] T 6fdf32e1756447c9a9e59956bb18b772 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:29.723096 5236 raft_consensus.cc:695] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:29.723845 5236 consensus_queue.cc:237] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.729979 5092 ts_tablet_manager.cc:1428] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.044s user 0.005s sys 0.001s
I20250626 19:57:29.730964 5092 tablet_bootstrap.cc:492] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:29.759497 4820 raft_consensus.cc:2466] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:29.770365 5091 tablet_bootstrap.cc:654] T 6fdf32e1756447c9a9e59956bb18b772 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.780934 5273 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ac80df82774b9b87c0be076b5637af. Latest consensus state: current_term: 6 leader_uuid: "62ac80df82774b9b87c0be076b5637af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:29.781662 5273 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:29.782804 5283 raft_consensus.cc:2802] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:29.783186 5283 raft_consensus.cc:491] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:29.783493 5283 raft_consensus.cc:3058] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:29.790514 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 89450.5us
I20250626 19:57:29.791707 5255 raft_consensus.cc:491] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:29.792323 5255 raft_consensus.cc:513] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.801756 5092 tablet_bootstrap.cc:654] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.804559 5255 leader_election.cc:290] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:29.805353 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7aee4118595346ef990a8a52eb007072" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:29.805935 4676 raft_consensus.cc:2466] T 7aee4118595346ef990a8a52eb007072 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:29.807017 4912 leader_election.cc:304] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:29.807039 5283 raft_consensus.cc:513] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.808008 5255 raft_consensus.cc:2802] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:29.808391 5255 raft_consensus.cc:491] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:29.808785 5255 raft_consensus.cc:3058] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.816529 5255 raft_consensus.cc:513] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.819761 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44960f8db08942879cde6f96e9c620da" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:29.820338 4674 raft_consensus.cc:3058] T 44960f8db08942879cde6f96e9c620da P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.824764 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7aee4118595346ef990a8a52eb007072" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:29.825595 4821 raft_consensus.cc:2466] T 7aee4118595346ef990a8a52eb007072 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:29.828374 4674 raft_consensus.cc:2466] T 44960f8db08942879cde6f96e9c620da P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:29.829587 4912 leader_election.cc:304] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:29.831063 5274 raft_consensus.cc:2802] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:29.831496 5274 raft_consensus.cc:695] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:29.832271 5274 consensus_queue.cc:237] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.834904 5275 consensus_queue.cc:1035] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:29.839658 5211 raft_consensus.cc:491] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:29.840095 5211 raft_consensus.cc:513] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.842865 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cb6bda529624ac5b5a87a03b4a02d56" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:29.843485 4675 raft_consensus.cc:2466] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:29.844866 5211 leader_election.cc:290] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:29.845430 4383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.190:33823, user_credentials={real_user=slave}} blocked reactor thread for 46934.7us
I20250626 19:57:29.845718 4912 leader_election.cc:304] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:29.850780 5275 raft_consensus.cc:2802] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:29.851168 5275 raft_consensus.cc:491] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:29.851485 5275 raft_consensus.cc:3058] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.854418 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cb6bda529624ac5b5a87a03b4a02d56" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:29.855212 4821 raft_consensus.cc:2466] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:29.864464 5275 raft_consensus.cc:513] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.866274 5275 leader_election.cc:290] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:29.867844 5091 tablet_bootstrap.cc:492] T 6fdf32e1756447c9a9e59956bb18b772 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:29.868506 5091 ts_tablet_manager.cc:1397] T 6fdf32e1756447c9a9e59956bb18b772 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.146s user 0.012s sys 0.000s
I20250626 19:57:29.868875 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cb6bda529624ac5b5a87a03b4a02d56" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:29.869916 4676 raft_consensus.cc:3058] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:29.872277 4383 consensus_peers.cc:487] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af -> Peer e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823): Couldn't send request to peer e6c58006b6be41c1bd6bdb5eb010d360. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:29.877231 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7aee4118595346ef990a8a52eb007072" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:29.877858 4675 raft_consensus.cc:3058] T 7aee4118595346ef990a8a52eb007072 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.879361 5255 leader_election.cc:290] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:29.880447 5283 leader_election.cc:290] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:29.897071 5247 raft_consensus.cc:491] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:29.897627 5247 raft_consensus.cc:513] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.899959 5247 leader_election.cc:290] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:29.901389 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e453a4f8ca44f4a940928d9f4fcd5be" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:29.902179 4675 raft_consensus.cc:2466] T 7aee4118595346ef990a8a52eb007072 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:29.903470 4912 leader_election.cc:304] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:29.904410 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e453a4f8ca44f4a940928d9f4fcd5be" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:29.905321 5283 raft_consensus.cc:2802] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:29.905783 5283 raft_consensus.cc:695] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:29.906531 5283 consensus_queue.cc:237] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.908694 4676 raft_consensus.cc:2466] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
W20250626 19:57:29.910131 4755 leader_election.cc:343] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:29.911136 4756 leader_election.cc:343] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:29.911468 4756 leader_election.cc:304] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:29.912079 5247 raft_consensus.cc:2747] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:29.910601 4912 leader_election.cc:304] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:29.912922 5255 raft_consensus.cc:2802] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:29.913280 5255 raft_consensus.cc:695] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:29.913918 5255 consensus_queue.cc:237] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:29.920680 5273 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "62ac80df82774b9b87c0be076b5637af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:29.921355 5273 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:29.933861 5092 tablet_bootstrap.cc:492] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:29.934269 5092 ts_tablet_manager.cc:1397] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.203s user 0.007s sys 0.006s
I20250626 19:57:29.935771 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7aee4118595346ef990a8a52eb007072" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:29.936352 4821 raft_consensus.cc:3058] T 7aee4118595346ef990a8a52eb007072 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.936661 5092 raft_consensus.cc:357] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.937254 5092 raft_consensus.cc:383] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.937500 5092 raft_consensus.cc:738] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.938040 5092 consensus_queue.cc:260] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.940311 5092 ts_tablet_manager.cc:1428] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250626 19:57:29.941130 5092 tablet_bootstrap.cc:492] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:29.943452 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cb6bda529624ac5b5a87a03b4a02d56" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:29.944100 4820 raft_consensus.cc:3058] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.945070 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44960f8db08942879cde6f96e9c620da" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:29.945583 4819 raft_consensus.cc:3058] T 44960f8db08942879cde6f96e9c620da P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:29.943611 4821 raft_consensus.cc:2466] T 7aee4118595346ef990a8a52eb007072 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:29.950659 5091 raft_consensus.cc:357] T 6fdf32e1756447c9a9e59956bb18b772 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.951323 5091 raft_consensus.cc:383] T 6fdf32e1756447c9a9e59956bb18b772 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.951561 5091 raft_consensus.cc:738] T 6fdf32e1756447c9a9e59956bb18b772 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.953783 5092 tablet_bootstrap.cc:654] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:29.960433 4819 raft_consensus.cc:2466] T 44960f8db08942879cde6f96e9c620da P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:29.962209 4820 raft_consensus.cc:2466] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:29.966380 5247 consensus_queue.cc:1035] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:29.967131 5091 consensus_queue.cc:260] T 6fdf32e1756447c9a9e59956bb18b772 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.967890 5092 tablet_bootstrap.cc:492] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:29.968329 5092 ts_tablet_manager.cc:1397] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.002s
I20250626 19:57:29.971160 5092 raft_consensus.cc:357] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.971988 5092 raft_consensus.cc:383] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:29.972229 5092 raft_consensus.cc:738] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:29.977938 5091 ts_tablet_manager.cc:1428] T 6fdf32e1756447c9a9e59956bb18b772 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.108s user 0.005s sys 0.000s
I20250626 19:57:29.978845 5091 tablet_bootstrap.cc:492] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:29.978782 5092 consensus_queue.cc:260] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:29.984057 5092 ts_tablet_manager.cc:1428] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.015s user 0.005s sys 0.002s
I20250626 19:57:30.035909 5091 tablet_bootstrap.cc:654] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:30.039465 4674 raft_consensus.cc:3058] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:30.039853 5244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "62ac80df82774b9b87c0be076b5637af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:30.040540 5244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: This master's current role is: LEADER
I20250626 19:57:30.052081 5251 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 19:57:30.067528 5236 raft_consensus.cc:491] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:30.068027 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_b2fa45f3c65e4e56b7617277fe9f16ca [id=048633318f5c4de0bd3a11f63c3f8103]
I20250626 19:57:30.068079 5236 raft_consensus.cc:513] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:30.069775 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_f579d2c76b104fc88466bcdcd54d2603 [id=049afd5ef383451fa96431d71b4f2ae4]
I20250626 19:57:30.072253 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_889f18064aab4774b17bfa478a0a7e33 [id=0df063bc889246c3ba841b1b2f8e6104]
I20250626 19:57:30.072563 5236 leader_election.cc:290] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:30.074515 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f91a78f6d474133976242ea8d259e29" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:30.074797 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_1a93e36798c7434da2c9cd04201320cd [id=21df672a819243f7a3d16bd6d0a599ac]
I20250626 19:57:30.074992 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f91a78f6d474133976242ea8d259e29" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:30.075582 4979 raft_consensus.cc:2466] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:30.077412 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_65941c5740784059b05dacd5e2f1a5c3 [id=3557609ecf024c1f91b7846221fa8112]
I20250626 19:57:30.077966 4611 leader_election.cc:304] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters: 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:30.079061 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_ad656f771f554bc98b2dc4db559e3acb [id=3c86bc87a4a64fd8af0af50c516a2242]
I20250626 19:57:30.079159 5176 raft_consensus.cc:2802] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 19:57:30.079479 5176 raft_consensus.cc:491] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:30.079808 5176 raft_consensus.cc:3058] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:57:30.080909 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_2ab026a162784703bf424759f2114968 [id=4419bc765f3a4ff1a104bb656bf642ea]
I20250626 19:57:30.082652 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]
I20250626 19:57:30.084132 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_689b779650c544c7baa707961e76e953 [id=49ced2ddeecf4d18a8693950b66b1e5c]
I20250626 19:57:30.086159 5247 consensus_queue.cc:1035] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:30.092067 5247 consensus_queue.cc:1035] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:30.097601 5247 consensus_queue.cc:1035] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:30.102205 5247 consensus_queue.cc:1035] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:30.107105 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_01a5eaed994e4495bd80c775e1a081a3 [id=52c0d515566c43b08a572196f917afdb]
I20250626 19:57:30.110551 5303 consensus_queue.cc:1035] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:30.112583 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_fdb231070332470b994fc496021fde25 [id=629ab240ba9a40e3a581b74ae5ed4f71]
I20250626 19:57:30.118261 5091 tablet_bootstrap.cc:492] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:30.118695 5091 ts_tablet_manager.cc:1397] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.140s user 0.009s sys 0.000s
I20250626 19:57:30.120916 5091 raft_consensus.cc:357] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:30.121527 5091 raft_consensus.cc:383] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:30.121760 5091 raft_consensus.cc:738] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:30.122292 5091 consensus_queue.cc:260] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:30.127995 5091 ts_tablet_manager.cc:1428] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250626 19:57:30.138633 5255 raft_consensus.cc:491] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:30.139137 5255 raft_consensus.cc:513] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:30.141162 5255 leader_election.cc:290] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:30.143908 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e453a4f8ca44f4a940928d9f4fcd5be" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:30.144624 4677 raft_consensus.cc:2466] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:30.148634 5274 raft_consensus.cc:491] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:30.149206 5274 raft_consensus.cc:513] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:30.151489 5274 leader_election.cc:290] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:30.153491 4912 leader_election.cc:304] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:30.154573 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fdf32e1756447c9a9e59956bb18b772" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:30.155722 4677 raft_consensus.cc:2466] T 6fdf32e1756447c9a9e59956bb18b772 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:30.158964 5274 raft_consensus.cc:2802] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:30.158288 5176 raft_consensus.cc:513] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:30.159447 5274 raft_consensus.cc:491] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:30.159796 5274 raft_consensus.cc:3058] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:30.159901 4676 raft_consensus.cc:1238] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 2 FOLLOWER]: Rejecting Update request from peer 93f885c6802341c889a40e9c20fbfff1 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250626 19:57:30.167085 5176 leader_election.cc:290] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 2 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:30.168354 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f91a78f6d474133976242ea8d259e29" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:30.176450 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_beec4e8c60ae4e6880c0ff709d3c0ff3 [id=6e84598eef744e8c9b1d918036bc67ec]
I20250626 19:57:30.178267 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_579a1ea24d1744068526a0765d5966ab [id=791d0e0df1e84922a9901aa6c456367b]
I20250626 19:57:30.174813 4912 leader_election.cc:304] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:30.179608 5255 raft_consensus.cc:2802] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:30.179673 5299 consensus_queue.cc:1046] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:30.180027 5255 raft_consensus.cc:491] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:30.180383 5255 raft_consensus.cc:3058] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:30.180548 5299 raft_consensus.cc:3053] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 19:57:30.180840 5299 raft_consensus.cc:738] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:30.181576 5299 consensus_queue.cc:260] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:30.183024 5299 raft_consensus.cc:3058] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:57:30.174876 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f91a78f6d474133976242ea8d259e29" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:30.187984 4611 leader_election.cc:304] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:30.189334 5176 raft_consensus.cc:2747] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 19:57:30.190810 5255 raft_consensus.cc:513] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:30.166719 5274 raft_consensus.cc:513] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:30.195173 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fdf32e1756447c9a9e59956bb18b772" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:30.195770 4676 raft_consensus.cc:3058] T 6fdf32e1756447c9a9e59956bb18b772 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:30.198302 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e453a4f8ca44f4a940928d9f4fcd5be" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:30.198879 4677 raft_consensus.cc:3058] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:30.204756 4676 raft_consensus.cc:2466] T 6fdf32e1756447c9a9e59956bb18b772 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:30.216210 5274 leader_election.cc:290] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:30.217212 5255 leader_election.cc:290] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:30.217901 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_27bcd169607a45599dc5a644e6ea5dd7 [id=8df233b9523c49adb3d24727bda24b9e]
I20250626 19:57:30.219491 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]
I20250626 19:57:30.222453 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_bd7be9809f0d462380d65cb6ed3f8c1f [id=9e99b7ec61654e36830933bab686b355]
I20250626 19:57:30.223902 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]
I20250626 19:57:30.225323 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_126ea31ac2c2483e84cedd411a0247b4 [id=ae81e1731bc1461385bf84e443ff0730]
I20250626 19:57:30.227344 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]
I20250626 19:57:30.228798 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_443891de930144a484281a0abf7d60f2 [id=b5086ce274d44f4dae848455c6274ec8]
I20250626 19:57:30.230437 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_e25c8c1da59041118a0072aa639e3771 [id=bea62f96f1cd4c2ab35704acf7def541]
I20250626 19:57:30.235500 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e453a4f8ca44f4a940928d9f4fcd5be" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:30.236173 4819 raft_consensus.cc:2466] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:30.237445 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fdf32e1756447c9a9e59956bb18b772" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:30.238026 4819 raft_consensus.cc:2466] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:30.239284 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e453a4f8ca44f4a940928d9f4fcd5be" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:30.239881 4819 raft_consensus.cc:3058] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:30.240998 5251 catalog_manager.cc:671] Loaded metadata for table default.Table_d23ee99af7b845b78d2200b962f681a6 [id=fc725d6df6774d4e922e2d98d461b111]
I20250626 19:57:30.251325 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fdf32e1756447c9a9e59956bb18b772" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:30.251909 4820 raft_consensus.cc:3058] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:30.205828 4677 raft_consensus.cc:2466] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:30.258308 4820 raft_consensus.cc:2466] T 6fdf32e1756447c9a9e59956bb18b772 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:30.260108 4912 leader_election.cc:304] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:30.260952 4912 leader_election.cc:304] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:30.262367 5255 raft_consensus.cc:2802] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:30.262823 5255 raft_consensus.cc:695] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:30.263589 5255 consensus_queue.cc:237] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:30.246584 4819 raft_consensus.cc:2466] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:30.266387 5274 raft_consensus.cc:2802] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:30.266858 5274 raft_consensus.cc:695] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:30.267652 5274 consensus_queue.cc:237] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:30.275713 5251 tablet_loader.cc:96] loaded metadata for tablet 04b0f44c1b564fdf98408881678a90a1 (table default.Table_fdb231070332470b994fc496021fde25 [id=629ab240ba9a40e3a581b74ae5ed4f71])
I20250626 19:57:30.285521 5251 tablet_loader.cc:96] loaded metadata for tablet 1cb6bda529624ac5b5a87a03b4a02d56 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091])
I20250626 19:57:30.313058 5251 tablet_loader.cc:96] loaded metadata for tablet 1e453a4f8ca44f4a940928d9f4fcd5be (table default.Table_f579d2c76b104fc88466bcdcd54d2603 [id=049afd5ef383451fa96431d71b4f2ae4])
I20250626 19:57:30.315951 5251 tablet_loader.cc:96] loaded metadata for tablet 2ab0320af47049cca3ca87172b9fe5e5 (table default.Table_d23ee99af7b845b78d2200b962f681a6 [id=fc725d6df6774d4e922e2d98d461b111])
I20250626 19:57:30.317287 5251 tablet_loader.cc:96] loaded metadata for tablet 3bd8dae5699c4725915814044d76a257 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194])
I20250626 19:57:30.318727 5251 tablet_loader.cc:96] loaded metadata for tablet 3cc96a3e9c174cbca0196c5a1b27455c (table default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9])
I20250626 19:57:30.320013 5251 tablet_loader.cc:96] loaded metadata for tablet 44960f8db08942879cde6f96e9c620da (table default.Table_27bcd169607a45599dc5a644e6ea5dd7 [id=8df233b9523c49adb3d24727bda24b9e])
I20250626 19:57:30.321259 5251 tablet_loader.cc:96] loaded metadata for tablet 499f6ae3262940b4859c14a53722482f (table default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1])
I20250626 19:57:30.322494 5251 tablet_loader.cc:96] loaded metadata for tablet 4b3bed19e7e34d24b995840a3bde5ef0 (table default.Table_ad656f771f554bc98b2dc4db559e3acb [id=3c86bc87a4a64fd8af0af50c516a2242])
I20250626 19:57:30.323836 5251 tablet_loader.cc:96] loaded metadata for tablet 4d8ffa9b658c413fa06d02ee6635dd1b (table default.Table_2ab026a162784703bf424759f2114968 [id=4419bc765f3a4ff1a104bb656bf642ea])
W20250626 19:57:30.328904 5309 log.cc:927] Time spent T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20250626 19:57:30.330457 5251 tablet_loader.cc:96] loaded metadata for tablet 6fdf32e1756447c9a9e59956bb18b772 (table default.Table_689b779650c544c7baa707961e76e953 [id=49ced2ddeecf4d18a8693950b66b1e5c])
I20250626 19:57:30.331698 5251 tablet_loader.cc:96] loaded metadata for tablet 76a2aa555b7a4aad86d923565436f17e (table default.Table_01a5eaed994e4495bd80c775e1a081a3 [id=52c0d515566c43b08a572196f917afdb])
I20250626 19:57:30.333745 5251 tablet_loader.cc:96] loaded metadata for tablet 7aee4118595346ef990a8a52eb007072 (table default.Table_443891de930144a484281a0abf7d60f2 [id=b5086ce274d44f4dae848455c6274ec8])
I20250626 19:57:30.334860 5251 tablet_loader.cc:96] loaded metadata for tablet 8f1a296c114b4149bc694f6b08a1f2e9 (table default.Table_bd7be9809f0d462380d65cb6ed3f8c1f [id=9e99b7ec61654e36830933bab686b355])
I20250626 19:57:30.335899 5251 tablet_loader.cc:96] loaded metadata for tablet 94327cab01df4338aaec157736d106f6 (table default.Table_beec4e8c60ae4e6880c0ff709d3c0ff3 [id=6e84598eef744e8c9b1d918036bc67ec])
I20250626 19:57:30.336962 5251 tablet_loader.cc:96] loaded metadata for tablet 9e4f889f0330432d9a8ec613cb541574 (table default.Table_65941c5740784059b05dacd5e2f1a5c3 [id=3557609ecf024c1f91b7846221fa8112])
I20250626 19:57:30.338281 5251 tablet_loader.cc:96] loaded metadata for tablet 9f91a78f6d474133976242ea8d259e29 (table default.Table_b2fa45f3c65e4e56b7617277fe9f16ca [id=048633318f5c4de0bd3a11f63c3f8103])
I20250626 19:57:30.339486 5251 tablet_loader.cc:96] loaded metadata for tablet a3b05549a1434a1fa3d337fdfb96a131 (table default.Table_579a1ea24d1744068526a0765d5966ab [id=791d0e0df1e84922a9901aa6c456367b])
I20250626 19:57:30.341482 5251 tablet_loader.cc:96] loaded metadata for tablet af58f34e2248432fae5c084bd3388028 (table default.Table_e25c8c1da59041118a0072aa639e3771 [id=bea62f96f1cd4c2ab35704acf7def541])
I20250626 19:57:30.344761 5251 tablet_loader.cc:96] loaded metadata for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 (table default.Table_889f18064aab4774b17bfa478a0a7e33 [id=0df063bc889246c3ba841b1b2f8e6104])
I20250626 19:57:30.346657 5251 tablet_loader.cc:96] loaded metadata for tablet df3da12971654d9abc2ee8840c4f1592 (table default.Table_1a93e36798c7434da2c9cd04201320cd [id=21df672a819243f7a3d16bd6d0a599ac])
I20250626 19:57:30.349349 5251 tablet_loader.cc:96] loaded metadata for tablet fa93fa6c652f4aca848e1e4910fb8175 (table default.Table_126ea31ac2c2483e84cedd411a0247b4 [id=ae81e1731bc1461385bf84e443ff0730])
I20250626 19:57:30.350682 5251 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 19:57:30.364231 5251 catalog_manager.cc:1261] Loaded cluster ID: 845ffa7fa66f40c8abadc28aed775425
I20250626 19:57:30.364609 5251 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 19:57:30.373567 5251 catalog_manager.cc:1506] Loading token signing keys...
I20250626 19:57:30.379362 5274 raft_consensus.cc:491] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:30.379830 5274 raft_consensus.cc:513] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:30.381548 5274 leader_election.cc:290] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:30.383607 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d8ffa9b658c413fa06d02ee6635dd1b" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:30.384311 4820 raft_consensus.cc:2391] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:30.385504 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d8ffa9b658c413fa06d02ee6635dd1b" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:30.387645 4912 leader_election.cc:304] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:30.388722 5274 raft_consensus.cc:3058] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:30.395336 5274 raft_consensus.cc:2747] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:30.397775 5251 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Loaded TSK: 0
I20250626 19:57:30.399470 5251 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 19:57:30.407023 5273 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 62ac80df82774b9b87c0be076b5637af)
I20250626 19:57:30.407567 5273 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:30.409621 5273 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:30.412318 4407 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 7 candidate_status { last_received { term: 6 index: 86 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:30.432631 4397 master_service.cc:948] ReplaceTablet: received request to replace tablet ea7374ad6ba64e46b7df9852fa33f145 from {username='slave'} at 127.0.0.1:34796
W20250626 19:57:30.437330 4462 leader_election.cc:336] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823): Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection refused (error 111)
I20250626 19:57:30.437794 4462 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31; no voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:30.438496 5273 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 19:57:30.438375 5176 consensus_queue.cc:1035] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:30.446666 4392 master_service.cc:948] ReplaceTablet: received request to replace tablet fa93fa6c652f4aca848e1e4910fb8175 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:30.471015 5274 consensus_queue.cc:1035] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:30.475783 5236 consensus_queue.cc:1035] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:30.439627 4398 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_443891de930144a484281a0abf7d60f2" } new_table_name: "default.Table_7c1a332875ad49abb1f0f16d7a027691" modify_external_catalogs: true
I20250626 19:57:30.444489 4395 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_a6836de37d3b41c7aa82c0c1e14ab4b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:30.446131 4393 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_65941c5740784059b05dacd5e2f1a5c3" } modify_external_catalogs: true
I20250626 19:57:30.495396 4393 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_65941c5740784059b05dacd5e2f1a5c3" } modify_external_catalogs: true
I20250626 19:57:30.441123 4399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_0b702c28aabb41ea8cd8b1c5e8ecef96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:30.499060 4399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b702c28aabb41ea8cd8b1c5e8ecef96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:30.442281 4396 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_4074ca47bf294632b7fe7f9958389a45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:30.504284 4396 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4074ca47bf294632b7fe7f9958389a45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:30.445308 4394 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_fdb231070332470b994fc496021fde25" } new_table_name: "default.Table_c98cdc17c95f4f1295e667b2c8324085" modify_external_catalogs: true
I20250626 19:57:30.450385 4391 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_b401c8a5ab6744029f693b1b5e0f306d" } modify_external_catalogs: true
I20250626 19:57:30.509380 4391 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_b401c8a5ab6744029f693b1b5e0f306d" } modify_external_catalogs: true
I20250626 19:57:30.513227 4391 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_1a93e36798c7434da2c9cd04201320cd" } modify_external_catalogs: true
I20250626 19:57:30.513760 4391 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_1a93e36798c7434da2c9cd04201320cd" } modify_external_catalogs: true
I20250626 19:57:30.436897 4397 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_5195fbf1909043f8b3fbbe59f976e0c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:30.518267 4395 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6836de37d3b41c7aa82c0c1e14ab4b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:57:30.520422 4397 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5195fbf1909043f8b3fbbe59f976e0c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:30.532752 5274 consensus_queue.cc:1035] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:30.555392 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:30.581251 5176 consensus_queue.cc:1035] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:30.593518 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet 512dc911bfce49b29c6fc8cd90523723 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:30.594461 5176 consensus_queue.cc:1035] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:57:30.627549 4398 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7aee4118595346ef990a8a52eb007072 consensus state has no leader
I20250626 19:57:30.628459 5176 consensus_queue.cc:1035] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:30.671294 5325 tablet_replica.cc:331] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:30.672438 5325 raft_consensus.cc:2241] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:30.673092 5325 raft_consensus.cc:2270] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 19:57:30.687575 4385 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 7aee4118595346ef990a8a52eb007072 (table default.Table_7c1a332875ad49abb1f0f16d7a027691 [id=b5086ce274d44f4dae848455c6274ec8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7aee4118595346ef990a8a52eb007072 consensus state has no leader
I20250626 19:57:30.696113 4393 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 9e4f889f0330432d9a8ec613cb541574
I20250626 19:57:30.696841 4393 master_service.cc:948] ReplaceTablet: received request to replace tablet 3bd8dae5699c4725915814044d76a257 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:30.680672 5325 ts_tablet_manager.cc:1905] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:30.728452 5211 consensus_queue.cc:1035] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:30.741180 5176 consensus_queue.cc:1035] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:30.780895 5255 consensus_queue.cc:1035] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:57:30.807597 4394 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04b0f44c1b564fdf98408881678a90a1 has no consensus state
I20250626 19:57:30.822328 5325 ts_tablet_manager.cc:1918] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:30.824785 5325 log.cc:1199] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/512dc911bfce49b29c6fc8cd90523723
I20250626 19:57:30.829624 5325 ts_tablet_manager.cc:1939] T 512dc911bfce49b29c6fc8cd90523723 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:30.833348 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 512dc911bfce49b29c6fc8cd90523723 (table default.Table_89c4b3541e7c4b08b25f28abf471d42d [id=f0d69891ab4745dbb68ea3c529d1a107]) successfully deleted
I20250626 19:57:30.856319 5255 consensus_queue.cc:1035] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:30.861747 4395 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_579a1ea24d1744068526a0765d5966ab" } new_table_name: "default.Table_38472096f51645c8b48cf4841ee73622" modify_external_catalogs: true
I20250626 19:57:30.888309 5255 consensus_queue.cc:1035] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:30.914582 4392 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet fa93fa6c652f4aca848e1e4910fb8175
I20250626 19:57:30.920636 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet fa93fa6c652f4aca848e1e4910fb8175 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fda659c30c44c8ba402ac5c7f3598e0) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:30.921401 5325 tablet_replica.cc:331] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:30.922658 4392 catalog_manager.cc:6652] ReplaceTablet: tablet fa93fa6c652f4aca848e1e4910fb8175 deleted and replaced by tablet 2fda659c30c44c8ba402ac5c7f3598e0
I20250626 19:57:30.922991 5325 raft_consensus.cc:2241] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:30.925174 5325 raft_consensus.cc:2270] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:30.927213 5325 ts_tablet_manager.cc:1905] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:30.934291 4383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.188:40801, user_credentials={real_user=slave}} blocked reactor thread for 43964.3us
W20250626 19:57:30.942952 4611 consensus_peers.cc:487] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:30.946123 4392 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_0b702c28aabb41ea8cd8b1c5e8ecef96" } modify_external_catalogs: true
I20250626 19:57:30.946686 4392 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_0b702c28aabb41ea8cd8b1c5e8ecef96" } modify_external_catalogs: true
I20250626 19:57:30.951347 4394 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_fbd8a487fac74f3087145875222b3dab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:30.953018 4394 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbd8a487fac74f3087145875222b3dab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:30.968209 5325 ts_tablet_manager.cc:1918] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:30.968689 5325 log.cc:1199] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/fa93fa6c652f4aca848e1e4910fb8175
I20250626 19:57:30.971175 5325 ts_tablet_manager.cc:1939] T fa93fa6c652f4aca848e1e4910fb8175 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:30.975680 4399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_9db5147373854d58a239e5e5b6048c72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:30.977141 4399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9db5147373854d58a239e5e5b6048c72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:30.980437 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet fa93fa6c652f4aca848e1e4910fb8175 (table default.Table_126ea31ac2c2483e84cedd411a0247b4 [id=ae81e1731bc1461385bf84e443ff0730]) successfully deleted
W20250626 19:57:30.987149 4395 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3b05549a1434a1fa3d337fdfb96a131 has no consensus state
I20250626 19:57:30.997692 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet fa93fa6c652f4aca848e1e4910fb8175 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fda659c30c44c8ba402ac5c7f3598e0) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:31.008507 5348 tablet_replica.cc:331] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:31.009608 5348 raft_consensus.cc:2241] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:31.011440 4800 tablet_service.cc:1515] Processing DeleteTablet for tablet fa93fa6c652f4aca848e1e4910fb8175 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fda659c30c44c8ba402ac5c7f3598e0) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:31.016570 5348 raft_consensus.cc:2270] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:31.023308 4395 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_2ab026a162784703bf424759f2114968" } new_table_name: "default.Table_681222687426414bb0838e947d315979" modify_external_catalogs: true
I20250626 19:57:31.024402 5348 ts_tablet_manager.cc:1905] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:31.027346 4397 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_d6eed56e9d514833bfe29ff3aba9718a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:31.028366 5349 tablet_replica.cc:331] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
W20250626 19:57:31.028795 4397 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6eed56e9d514833bfe29ff3aba9718a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:31.029320 5349 raft_consensus.cc:2241] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:31.029925 5349 raft_consensus.cc:2270] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:31.044682 5349 ts_tablet_manager.cc:1905] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:31.046576 4957 tablet_service.cc:1468] Processing CreateTablet for tablet 2ab0320af47049cca3ca87172b9fe5e5 (DEFAULT_TABLE table=default.Table_d23ee99af7b845b78d2200b962f681a6 [id=fc725d6df6774d4e922e2d98d461b111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.047698 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ab0320af47049cca3ca87172b9fe5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.051977 4652 tablet_service.cc:1468] Processing CreateTablet for tablet 2ab0320af47049cca3ca87172b9fe5e5 (DEFAULT_TABLE table=default.Table_d23ee99af7b845b78d2200b962f681a6 [id=fc725d6df6774d4e922e2d98d461b111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.053036 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ab0320af47049cca3ca87172b9fe5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.057926 4657 tablet_service.cc:1468] Processing CreateTablet for tablet a3b05549a1434a1fa3d337fdfb96a131 (DEFAULT_TABLE table=default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.059101 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3b05549a1434a1fa3d337fdfb96a131. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.064831 5348 ts_tablet_manager.cc:1918] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:31.065297 5348 log.cc:1199] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/fa93fa6c652f4aca848e1e4910fb8175
I20250626 19:57:31.066114 5348 ts_tablet_manager.cc:1939] T fa93fa6c652f4aca848e1e4910fb8175 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:31.069052 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet fa93fa6c652f4aca848e1e4910fb8175 (table default.Table_126ea31ac2c2483e84cedd411a0247b4 [id=ae81e1731bc1461385bf84e443ff0730]) successfully deleted
I20250626 19:57:31.073433 4956 tablet_service.cc:1468] Processing CreateTablet for tablet a3b05549a1434a1fa3d337fdfb96a131 (DEFAULT_TABLE table=default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.074538 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3b05549a1434a1fa3d337fdfb96a131. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.074996 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 8f1a296c114b4149bc694f6b08a1f2e9 (DEFAULT_TABLE table=default.Table_bd7be9809f0d462380d65cb6ed3f8c1f [id=9e99b7ec61654e36830933bab686b355]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.076051 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1a296c114b4149bc694f6b08a1f2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.077044 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 8f1a296c114b4149bc694f6b08a1f2e9 (DEFAULT_TABLE table=default.Table_bd7be9809f0d462380d65cb6ed3f8c1f [id=9e99b7ec61654e36830933bab686b355]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.077983 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1a296c114b4149bc694f6b08a1f2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.086987 5211 consensus_queue.cc:1035] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:31.087231 4655 tablet_service.cc:1468] Processing CreateTablet for tablet 04b0f44c1b564fdf98408881678a90a1 (DEFAULT_TABLE table=default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.088289 4655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b0f44c1b564fdf98408881678a90a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.093392 5255 consensus_queue.cc:1035] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:31.110330 4958 tablet_service.cc:1468] Processing CreateTablet for tablet 04b0f44c1b564fdf98408881678a90a1 (DEFAULT_TABLE table=default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.111469 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b0f44c1b564fdf98408881678a90a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.111845 5350 tablet_bootstrap.cc:492] T a3b05549a1434a1fa3d337fdfb96a131 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:31.070006 4398 catalog_manager.cc:5582] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:31.123806 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 44960f8db08942879cde6f96e9c620da (table default.Table_27bcd169607a45599dc5a644e6ea5dd7 [id=8df233b9523c49adb3d24727bda24b9e]). Expected version 1 got 0
I20250626 19:57:31.125037 4398 catalog_manager.cc:5582] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:31.126410 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 1cb6bda529624ac5b5a87a03b4a02d56 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:31.118580 5353 tablet_bootstrap.cc:492] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:31.086095 5349 ts_tablet_manager.cc:1918] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:31.127200 5349 log.cc:1199] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/fa93fa6c652f4aca848e1e4910fb8175
I20250626 19:57:31.128085 5349 ts_tablet_manager.cc:1939] T fa93fa6c652f4aca848e1e4910fb8175 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:31.143564 5353 tablet_bootstrap.cc:654] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:31.144018 4654 tablet_service.cc:1468] Processing CreateTablet for tablet 76a2aa555b7a4aad86d923565436f17e (DEFAULT_TABLE table=default.Table_01a5eaed994e4495bd80c775e1a081a3 [id=52c0d515566c43b08a572196f917afdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.145162 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a2aa555b7a4aad86d923565436f17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.148197 5350 tablet_bootstrap.cc:654] T a3b05549a1434a1fa3d337fdfb96a131 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:31.152917 5255 consensus_queue.cc:1035] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:31.162712 5255 consensus_queue.cc:1035] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:31.185079 4956 tablet_service.cc:1468] Processing CreateTablet for tablet 76a2aa555b7a4aad86d923565436f17e (DEFAULT_TABLE table=default.Table_01a5eaed994e4495bd80c775e1a081a3 [id=52c0d515566c43b08a572196f917afdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.185357 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 94327cab01df4338aaec157736d106f6 (DEFAULT_TABLE table=default.Table_beec4e8c60ae4e6880c0ff709d3c0ff3 [id=6e84598eef744e8c9b1d918036bc67ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.186650 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94327cab01df4338aaec157736d106f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.187531 4954 tablet_service.cc:1468] Processing CreateTablet for tablet 94327cab01df4338aaec157736d106f6 (DEFAULT_TABLE table=default.Table_beec4e8c60ae4e6880c0ff709d3c0ff3 [id=6e84598eef744e8c9b1d918036bc67ec]), partition=RANGE (key) PARTITION UNBOUNDED
W20250626 19:57:31.139994 4383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.188:40801, user_credentials={real_user=slave}} blocked reactor thread for 117561us
I20250626 19:57:31.192464 5350 tablet_bootstrap.cc:492] T a3b05549a1434a1fa3d337fdfb96a131 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:31.193799 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 2ab0320af47049cca3ca87172b9fe5e5 (DEFAULT_TABLE table=default.Table_d23ee99af7b845b78d2200b962f681a6 [id=fc725d6df6774d4e922e2d98d461b111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.194741 4798 tablet_service.cc:1468] Processing CreateTablet for tablet a3b05549a1434a1fa3d337fdfb96a131 (DEFAULT_TABLE table=default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.195154 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ab0320af47049cca3ca87172b9fe5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.195477 4797 tablet_service.cc:1468] Processing CreateTablet for tablet 8f1a296c114b4149bc694f6b08a1f2e9 (DEFAULT_TABLE table=default.Table_bd7be9809f0d462380d65cb6ed3f8c1f [id=9e99b7ec61654e36830933bab686b355]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.195794 4798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3b05549a1434a1fa3d337fdfb96a131. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.196583 4797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1a296c114b4149bc694f6b08a1f2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.200286 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 04b0f44c1b564fdf98408881678a90a1 (DEFAULT_TABLE table=default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.201380 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b0f44c1b564fdf98408881678a90a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.207074 5350 ts_tablet_manager.cc:1397] T a3b05549a1434a1fa3d337fdfb96a131 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.096s user 0.009s sys 0.004s
I20250626 19:57:31.215880 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 76a2aa555b7a4aad86d923565436f17e (DEFAULT_TABLE table=default.Table_01a5eaed994e4495bd80c775e1a081a3 [id=52c0d515566c43b08a572196f917afdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.217056 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a2aa555b7a4aad86d923565436f17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.216871 4796 tablet_service.cc:1468] Processing CreateTablet for tablet 94327cab01df4338aaec157736d106f6 (DEFAULT_TABLE table=default.Table_beec4e8c60ae4e6880c0ff709d3c0ff3 [id=6e84598eef744e8c9b1d918036bc67ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:31.218016 4796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94327cab01df4338aaec157736d106f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.210355 5350 raft_consensus.cc:357] T a3b05549a1434a1fa3d337fdfb96a131 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.228740 5350 raft_consensus.cc:383] T a3b05549a1434a1fa3d337fdfb96a131 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:31.229072 5350 raft_consensus.cc:738] T a3b05549a1434a1fa3d337fdfb96a131 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:31.235073 5255 consensus_queue.cc:1035] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:31.242059 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet fa93fa6c652f4aca848e1e4910fb8175 (table default.Table_126ea31ac2c2483e84cedd411a0247b4 [id=ae81e1731bc1461385bf84e443ff0730]) successfully deleted
I20250626 19:57:31.189699 4398 catalog_manager.cc:5582] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:31.244746 4398 catalog_manager.cc:5582] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } }
I20250626 19:57:31.229733 5350 consensus_queue.cc:260] T a3b05549a1434a1fa3d337fdfb96a131 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.252885 5350 ts_tablet_manager.cc:1428] T a3b05549a1434a1fa3d337fdfb96a131 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20250626 19:57:31.257202 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a2aa555b7a4aad86d923565436f17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.259282 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94327cab01df4338aaec157736d106f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:31.257808 5350 tablet_bootstrap.cc:492] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:31.264245 5353 tablet_bootstrap.cc:492] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:31.265164 5353 ts_tablet_manager.cc:1397] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.147s user 0.010s sys 0.017s
I20250626 19:57:31.272117 5350 tablet_bootstrap.cc:654] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:31.246165 4398 catalog_manager.cc:5582] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } }
I20250626 19:57:31.273294 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 3bd8dae5699c4725915814044d76a257 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:31.273213 5353 raft_consensus.cc:357] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.273887 5353 raft_consensus.cc:383] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:31.274466 4398 catalog_manager.cc:5582] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:31.274164 5353 raft_consensus.cc:738] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:31.285786 5353 consensus_queue.cc:260] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.286820 5366 tablet_bootstrap.cc:492] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
W20250626 19:57:31.289460 4383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.190:33823, user_credentials={real_user=slave}} blocked reactor thread for 46680.3us
I20250626 19:57:31.293769 5353 ts_tablet_manager.cc:1428] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.023s user 0.002s sys 0.004s
W20250626 19:57:31.279917 4611 consensus_peers.cc:487] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 512dc911bfce49b29c6fc8cd90523723. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:31.300527 5353 tablet_bootstrap.cc:492] T 8f1a296c114b4149bc694f6b08a1f2e9 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:31.306146 5366 tablet_bootstrap.cc:654] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:31.312148 5266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20250626 19:57:31.314411 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 7aee4118595346ef990a8a52eb007072 (table default.Table_7c1a332875ad49abb1f0f16d7a027691 [id=b5086ce274d44f4dae848455c6274ec8]). Expected version 1 got 0
I20250626 19:57:31.316921 5350 tablet_bootstrap.cc:492] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
W20250626 19:57:31.317658 5266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20250626 19:57:31.318077 5353 tablet_bootstrap.cc:654] T 8f1a296c114b4149bc694f6b08a1f2e9 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:31.327484 5266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250626 19:57:31.331028 5350 ts_tablet_manager.cc:1397] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.073s user 0.006s sys 0.005s
I20250626 19:57:31.334088 5350 raft_consensus.cc:357] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.319363 4392 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 7ed43a25e8954f18843793d56b029441
I20250626 19:57:31.342455 4392 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_4074ca47bf294632b7fe7f9958389a45" } new_table_name: "default.Table_ffa0f924726d484ab1a91bf90ed84f93" modify_external_catalogs: true
I20250626 19:57:31.322058 4394 master_service.cc:948] ReplaceTablet: received request to replace tablet afd1fafb53434957b002fad5c77431d4 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:31.322264 5255 consensus_queue.cc:1035] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:31.328864 4399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_fde5b6bd4ee8417e85ffccad3bd2a581"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:31.347432 4399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fde5b6bd4ee8417e85ffccad3bd2a581 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:31.366902 5350 raft_consensus.cc:383] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:31.367393 5350 raft_consensus.cc:738] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:31.357857 4398 catalog_manager.cc:5582] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:31.373452 5350 consensus_queue.cc:260] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.379079 5366 tablet_bootstrap.cc:492] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:31.379551 5366 ts_tablet_manager.cc:1397] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.093s user 0.013s sys 0.000s
I20250626 19:57:31.382262 5366 raft_consensus.cc:357] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.382869 4398 catalog_manager.cc:5582] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } }
I20250626 19:57:31.392769 5350 ts_tablet_manager.cc:1428] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.060s user 0.004s sys 0.001s
I20250626 19:57:31.396211 5350 tablet_bootstrap.cc:492] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:31.399051 5366 raft_consensus.cc:383] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:31.405191 5350 tablet_bootstrap.cc:654] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:31.405541 5366 raft_consensus.cc:738] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:31.413187 5366 consensus_queue.cc:260] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.417024 5366 ts_tablet_manager.cc:1428] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20250626 19:57:31.418915 5366 tablet_bootstrap.cc:492] T 76a2aa555b7a4aad86d923565436f17e P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:31.405896 4398 catalog_manager.cc:5582] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:31.425359 5366 tablet_bootstrap.cc:654] T 76a2aa555b7a4aad86d923565436f17e P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:31.427979 4391 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet df3da12971654d9abc2ee8840c4f1592
I20250626 19:57:31.431696 4956 tablet_service.cc:1515] Processing DeleteTablet for tablet df3da12971654d9abc2ee8840c4f1592 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:30 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:31.432485 5325 tablet_replica.cc:331] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:31.432482 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet df3da12971654d9abc2ee8840c4f1592 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:30 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:31.433140 5348 tablet_replica.cc:331] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:31.434070 5348 raft_consensus.cc:2241] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:31.434998 5348 raft_consensus.cc:2270] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:31.435078 5325 raft_consensus.cc:2241] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:31.435570 5325 raft_consensus.cc:2270] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:31.436961 5348 ts_tablet_manager.cc:1905] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:31.437381 5325 ts_tablet_manager.cc:1905] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:31.443854 4395 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d8ffa9b658c413fa06d02ee6635dd1b consensus state has no leader
I20250626 19:57:31.444948 5353 tablet_bootstrap.cc:492] T 8f1a296c114b4149bc694f6b08a1f2e9 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:31.445451 5353 ts_tablet_manager.cc:1397] T 8f1a296c114b4149bc694f6b08a1f2e9 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.145s user 0.016s sys 0.001s
I20250626 19:57:31.448101 5353 raft_consensus.cc:357] T 8f1a296c114b4149bc694f6b08a1f2e9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.448803 5350 tablet_bootstrap.cc:492] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:31.449266 5350 ts_tablet_manager.cc:1397] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.000s
W20250626 19:57:31.450230 4392 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c990e4eca5cf4a8e8295e9599cf5be29 has no consensus state
I20250626 19:57:31.452307 5350 raft_consensus.cc:357] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.453054 5350 raft_consensus.cc:383] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:31.453320 4797 tablet_service.cc:1515] Processing DeleteTablet for tablet df3da12971654d9abc2ee8840c4f1592 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:30 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:31.453359 5350 raft_consensus.cc:738] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:31.454016 5349 tablet_replica.cc:331] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:31.454916 5349 raft_consensus.cc:2241] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:31.455451 5349 raft_consensus.cc:2270] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:31.457566 5349 ts_tablet_manager.cc:1905] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:31.462862 5353 raft_consensus.cc:383] T 8f1a296c114b4149bc694f6b08a1f2e9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:31.468475 5353 raft_consensus.cc:738] T 8f1a296c114b4149bc694f6b08a1f2e9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:31.469552 5353 consensus_queue.cc:260] T 8f1a296c114b4149bc694f6b08a1f2e9 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.471120 5350 consensus_queue.cc:260] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.477591 5350 ts_tablet_manager.cc:1428] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20250626 19:57:31.482363 5350 tablet_bootstrap.cc:492] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:31.490000 5349 ts_tablet_manager.cc:1918] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250626 19:57:31.493095 5349 log.cc:1199] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/df3da12971654d9abc2ee8840c4f1592
I20250626 19:57:31.494446 5349 ts_tablet_manager.cc:1939] T df3da12971654d9abc2ee8840c4f1592 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:31.496625 5353 ts_tablet_manager.cc:1428] T 8f1a296c114b4149bc694f6b08a1f2e9 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.051s user 0.005s sys 0.000s
I20250626 19:57:31.497612 5350 tablet_bootstrap.cc:654] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:31.502118 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet df3da12971654d9abc2ee8840c4f1592 (table default.Table_1a93e36798c7434da2c9cd04201320cd [id=21df672a819243f7a3d16bd6d0a599ac]) successfully deleted
I20250626 19:57:31.512089 5350 tablet_bootstrap.cc:492] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:31.515205 5350 ts_tablet_manager.cc:1397] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.010s
I20250626 19:57:31.518256 5350 raft_consensus.cc:357] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.522279 5350 raft_consensus.cc:383] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:31.523008 5350 raft_consensus.cc:738] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:31.513800 5353 tablet_bootstrap.cc:492] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:31.524248 5350 consensus_queue.cc:260] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
W20250626 19:57:31.527547 4396 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 66) took 788 ms (client timeout 1000 ms). Trace:
W20250626 19:57:31.527916 4396 rpcz_store.cc:269] 0626 19:57:30.739117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:30.739260 (+ 143us) service_pool.cc:225] Handling call
0626 19:57:30.764655 (+ 25395us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe7915557ee 00007fe78f26c624 00007fe78f26c5b2 00007fe78f26c4e1 00007fe78f26c45c 00007fe78f26b712 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:31.527482 (+762827us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":54206720}
I20250626 19:57:31.530301 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:31.532284 5350 ts_tablet_manager.cc:1428] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
W20250626 19:57:31.536988 4390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 72) took 973 ms (client timeout 1000 ms). Trace:
I20250626 19:57:31.535935 5350 tablet_bootstrap.cc:492] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
W20250626 19:57:31.537410 4390 rpcz_store.cc:269] 0626 19:57:30.563735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:30.605000 (+ 41265us) service_pool.cc:225] Handling call
0626 19:57:30.645474 (+ 40474us) spinlock_profiling.cc:243] Waited 21.5 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe7915557ee 00007fe78f26c624 00007fe78f26c5b2 00007fe78f26c4e1 00007fe78f26c45c 00007fe78f26b712 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:31.536919 (+891445us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":49510784}
I20250626 19:57:31.539750 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
W20250626 19:57:31.541882 5266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 19:57:31.542248 5266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 19:57:31.542601 5266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 19:57:31.551029 5266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 19:57:31.554057 5325 ts_tablet_manager.cc:1918] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250626 19:57:31.554711 5325 log.cc:1199] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/df3da12971654d9abc2ee8840c4f1592
I20250626 19:57:31.555485 5325 ts_tablet_manager.cc:1939] T df3da12971654d9abc2ee8840c4f1592 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:31.558209 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet df3da12971654d9abc2ee8840c4f1592 (table default.Table_1a93e36798c7434da2c9cd04201320cd [id=21df672a819243f7a3d16bd6d0a599ac]) successfully deleted
I20250626 19:57:31.565511 5353 tablet_bootstrap.cc:654] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:31.571550 4383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.188:40801, user_credentials={real_user=slave}} blocked reactor thread for 56307.3us
I20250626 19:57:31.580545 5350 tablet_bootstrap.cc:654] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:31.585182 4394 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_a2a4a2c9c59248dfb73f7db218b0d982"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:31.587746 4399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_427be70c41324505a21930cf7e51915f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:31.589205 4399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_427be70c41324505a21930cf7e51915f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:31.592298 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:31.595844 4392 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_681222687426414bb0838e947d315979" } modify_external_catalogs: true
I20250626 19:57:31.596527 4392 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_681222687426414bb0838e947d315979" } modify_external_catalogs: true
W20250626 19:57:31.614254 4394 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2a4a2c9c59248dfb73f7db218b0d982 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:31.614849 5368 raft_consensus.cc:491] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:31.615290 5368 raft_consensus.cc:513] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.617803 5368 leader_election.cc:290] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:31.619654 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04b0f44c1b564fdf98408881678a90a1" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:31.619808 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04b0f44c1b564fdf98408881678a90a1" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:31.621111 4611 leader_election.cc:343] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:31.621563 4609 leader_election.cc:343] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:31.621907 4609 leader_election.cc:304] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:31.622476 5368 raft_consensus.cc:2747] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:31.632368 5350 tablet_bootstrap.cc:492] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:31.632834 5350 ts_tablet_manager.cc:1397] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.097s user 0.009s sys 0.001s
I20250626 19:57:31.635972 5350 raft_consensus.cc:357] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.636641 5350 raft_consensus.cc:383] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:31.636910 5350 raft_consensus.cc:738] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:31.637518 5350 consensus_queue.cc:260] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.651129 5350 ts_tablet_manager.cc:1428] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.018s user 0.001s sys 0.005s
W20250626 19:57:31.667538 4383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.190:33823, user_credentials={real_user=slave}} blocked reactor thread for 88978.3us
I20250626 19:57:31.671866 5350 tablet_bootstrap.cc:492] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:31.677245 5348 ts_tablet_manager.cc:1918] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250626 19:57:31.677838 5348 log.cc:1199] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/df3da12971654d9abc2ee8840c4f1592
I20250626 19:57:31.678969 5366 tablet_bootstrap.cc:492] T 76a2aa555b7a4aad86d923565436f17e P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:31.679414 5366 ts_tablet_manager.cc:1397] T 76a2aa555b7a4aad86d923565436f17e P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.261s user 0.025s sys 0.044s
I20250626 19:57:31.682152 5366 raft_consensus.cc:357] T 76a2aa555b7a4aad86d923565436f17e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.682817 5366 raft_consensus.cc:383] T 76a2aa555b7a4aad86d923565436f17e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:31.683115 5366 raft_consensus.cc:738] T 76a2aa555b7a4aad86d923565436f17e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:31.683817 5366 consensus_queue.cc:260] T 76a2aa555b7a4aad86d923565436f17e P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.684981 5266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.46.148:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.46.190:33823,127.4.46.189:45039,127.4.46.188:40801
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.190:33823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.46.190
--webserver_port=36587
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6785
TSAN enabled
I20250626 19:57:31.686681 5266 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 19:57:31.687186 5353 tablet_bootstrap.cc:492] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:31.688191 5353 ts_tablet_manager.cc:1397] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.175s user 0.013s sys 0.020s
I20250626 19:57:31.689026 5266 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 19:57:31.689354 5350 tablet_bootstrap.cc:654] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:31.690701 5353 raft_consensus.cc:357] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.691435 5353 raft_consensus.cc:383] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:31.691736 5353 raft_consensus.cc:738] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:31.693845 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_d0820205b84340099fad74c11e7ab51c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:31.696135 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0820205b84340099fad74c11e7ab51c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:31.697353 4397 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_f579d2c76b104fc88466bcdcd54d2603" } modify_external_catalogs: true
I20250626 19:57:31.678804 5348 ts_tablet_manager.cc:1939] T df3da12971654d9abc2ee8840c4f1592 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:31.701704 5303 raft_consensus.cc:491] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:31.702140 5303 raft_consensus.cc:513] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.703141 5350 tablet_bootstrap.cc:492] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:31.703545 5350 ts_tablet_manager.cc:1397] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.009s
I20250626 19:57:31.704130 5303 leader_election.cc:290] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:31.710538 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ab0320af47049cca3ca87172b9fe5e5" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:31.711197 4676 raft_consensus.cc:2466] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:31.712327 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ab0320af47049cca3ca87172b9fe5e5" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:31.706527 5350 raft_consensus.cc:357] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.715265 5350 raft_consensus.cc:383] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:31.715538 5350 raft_consensus.cc:738] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:31.716236 5350 consensus_queue.cc:260] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.723829 4755 leader_election.cc:304] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:31.725696 5360 raft_consensus.cc:2802] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:31.726028 5360 raft_consensus.cc:491] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:31.726307 5360 raft_consensus.cc:3058] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:31.741750 5360 raft_consensus.cc:513] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.743736 5360 leader_election.cc:290] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:31.745347 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ab0320af47049cca3ca87172b9fe5e5" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:31.745724 5350 ts_tablet_manager.cc:1428] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20250626 19:57:31.747265 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ab0320af47049cca3ca87172b9fe5e5" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:31.747483 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet df3da12971654d9abc2ee8840c4f1592 (table default.Table_1a93e36798c7434da2c9cd04201320cd [id=21df672a819243f7a3d16bd6d0a599ac]) successfully deleted
I20250626 19:57:31.747809 4676 raft_consensus.cc:3058] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:31.755209 4393 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 3bd8dae5699c4725915814044d76a257
I20250626 19:57:31.758270 4956 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bd8dae5699c4725915814044d76a257 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c0338c05a974e0f96865398722c5e8f) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:31.758858 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bd8dae5699c4725915814044d76a257 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c0338c05a974e0f96865398722c5e8f) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:31.759111 5325 tablet_replica.cc:331] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:31.759485 5348 tablet_replica.cc:331] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:31.759903 4393 catalog_manager.cc:6652] ReplaceTablet: tablet 3bd8dae5699c4725915814044d76a257 deleted and replaced by tablet 4c0338c05a974e0f96865398722c5e8f
I20250626 19:57:31.760246 4393 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34796 (request call id 45) took 1085 ms. Trace:
I20250626 19:57:31.760551 5348 raft_consensus.cc:2241] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:31.761409 5348 raft_consensus.cc:2270] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:31.763059 4397 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_f579d2c76b104fc88466bcdcd54d2603" } modify_external_catalogs: true
I20250626 19:57:31.763626 5325 raft_consensus.cc:2241] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:31.763695 5348 ts_tablet_manager.cc:1905] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:31.764123 5325 raft_consensus.cc:2270] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:31.760571 4393 rpcz_store.cc:276] 0626 19:57:30.674662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:30.696719 (+ 22057us) service_pool.cc:225] Handling call
0626 19:57:30.763820 (+ 67101us) spinlock_profiling.cc:243] Waited 65.7 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe791559891 00007fe78f2943a7 00007fe78f294332 00007fe78f294261 00007fe78f2941dc 00007fe78f293492 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:31.760190 (+996370us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:31.651575 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:31.653362 (+ 1787us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:31.653440 (+ 78us) write_op.cc:435] Acquired schema lock
0626 19:57:31.653464 (+ 24us) tablet.cc:660] Decoding operations
0626 19:57:31.653848 (+ 384us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:31.653899 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:31.653925 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:31.654131 (+ 206us) tablet.cc:699] Row locks acquired
0626 19:57:31.654144 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:57:31.654269 (+ 125us) write_op.cc:270] Start()
0626 19:57:31.654368 (+ 99us) write_op.cc:276] Timestamp: P: 1750967851654347 usec, L: 0
0626 19:57:31.654400 (+ 32us) op_driver.cc:348] REPLICATION: starting
0626 19:57:31.654964 (+ 564us) log.cc:844] Serialized 658 byte log entry
0626 19:57:31.730136 (+ 75172us) op_driver.cc:464] REPLICATION: finished
0626 19:57:31.730818 (+ 682us) write_op.cc:301] APPLY: starting
0626 19:57:31.730882 (+ 64us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:31.730994 (+ 112us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:31.731082 (+ 88us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:31.731410 (+ 328us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:31.731634 (+ 224us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:31.731675 (+ 41us) write_op.cc:312] APPLY: finished
0626 19:57:31.731857 (+ 182us) log.cc:844] Serialized 28 byte log entry
0626 19:57:31.732000 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:31.732065 (+ 65us) write_op.cc:454] Released schema lock
0626 19:57:31.732099 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:31.732353 (+ 254us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:31.732377 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":151158656,"child_traces":[["op",{"apply.queue_time_us":484,"apply.run_cpu_time_us":1533,"apply.run_wall_time_us":1532,"num_ops":2,"prepare.queue_time_us":126430,"prepare.run_cpu_time_us":2429,"prepare.run_wall_time_us":4891,"replication_time_us":75635}]]}
I20250626 19:57:31.766011 5325 ts_tablet_manager.cc:1905] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:31.766773 4393 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_27bcd169607a45599dc5a644e6ea5dd7" } new_table_name: "default.Table_eff15db1de2e4aa3977a66f4af2b3dcd" modify_external_catalogs: true
I20250626 19:57:31.780527 5366 ts_tablet_manager.cc:1428] T 76a2aa555b7a4aad86d923565436f17e P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.101s user 0.004s sys 0.002s
W20250626 19:57:31.794458 4384 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 4d8ffa9b658c413fa06d02ee6635dd1b (table default.Table_681222687426414bb0838e947d315979 [id=4419bc765f3a4ff1a104bb656bf642ea], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4d8ffa9b658c413fa06d02ee6635dd1b consensus state has no leader
I20250626 19:57:31.797818 5368 raft_consensus.cc:491] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:31.798355 5368 raft_consensus.cc:513] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.801146 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a2aa555b7a4aad86d923565436f17e" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:31.801767 4819 raft_consensus.cc:2466] T 76a2aa555b7a4aad86d923565436f17e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
W20250626 19:57:31.802138 5381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:57:31.802718 5368 leader_election.cc:290] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:31.802742 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a2aa555b7a4aad86d923565436f17e" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:31.803953 4609 leader_election.cc:304] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:31.804093 5348 ts_tablet_manager.cc:1918] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250626 19:57:31.804589 4611 leader_election.cc:343] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:31.804726 5383 raft_consensus.cc:2802] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:31.805069 5383 raft_consensus.cc:491] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:31.805397 5383 raft_consensus.cc:3058] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:31.814316 4797 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bd8dae5699c4725915814044d76a257 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c0338c05a974e0f96865398722c5e8f) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:31.815049 5349 tablet_replica.cc:331] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:31.815812 5349 raft_consensus.cc:2241] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:31.816277 5349 raft_consensus.cc:2270] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:31.817129 5353 consensus_queue.cc:260] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.818082 4978 raft_consensus.cc:2466] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:31.818194 5349 ts_tablet_manager.cc:1905] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:31.804451 5348 log.cc:1199] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/3bd8dae5699c4725915814044d76a257
I20250626 19:57:31.831037 5348 ts_tablet_manager.cc:1939] T 3bd8dae5699c4725915814044d76a257 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:31.843773 5389 tablet.cc:1722] T 7aee4118595346ef990a8a52eb007072 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:31.854492 5349 ts_tablet_manager.cc:1918] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:31.854625 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 3bd8dae5699c4725915814044d76a257 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]) successfully deleted
I20250626 19:57:31.858899 5368 raft_consensus.cc:491] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:31.859388 5368 raft_consensus.cc:513] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.860435 5383 raft_consensus.cc:513] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.862617 5383 leader_election.cc:290] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:31.863350 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f1a296c114b4149bc694f6b08a1f2e9" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:31.863677 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a2aa555b7a4aad86d923565436f17e" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:31.864450 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a2aa555b7a4aad86d923565436f17e" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:31.864929 4820 raft_consensus.cc:3058] T 76a2aa555b7a4aad86d923565436f17e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:31.865644 4676 raft_consensus.cc:2466] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:31.867223 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f1a296c114b4149bc694f6b08a1f2e9" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:31.867527 5366 tablet_bootstrap.cc:492] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:31.867823 4978 raft_consensus.cc:2466] T 8f1a296c114b4149bc694f6b08a1f2e9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
W20250626 19:57:31.868708 4609 leader_election.cc:343] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:31.869690 4611 leader_election.cc:343] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:31.871774 5368 leader_election.cc:290] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:31.872206 4611 leader_election.cc:304] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters: 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:31.872326 4820 raft_consensus.cc:2466] T 76a2aa555b7a4aad86d923565436f17e P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:31.873355 5383 raft_consensus.cc:2802] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:31.873518 4609 leader_election.cc:304] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:31.873697 5383 raft_consensus.cc:491] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:31.874039 5383 raft_consensus.cc:3058] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:31.874545 5384 raft_consensus.cc:491] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:31.874909 5349 log.cc:1199] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/3bd8dae5699c4725915814044d76a257
I20250626 19:57:31.875001 5384 raft_consensus.cc:513] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.875679 5349 ts_tablet_manager.cc:1939] T 3bd8dae5699c4725915814044d76a257 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:31.875891 5368 raft_consensus.cc:2802] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:31.877066 5384 leader_election.cc:290] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:31.880112 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94327cab01df4338aaec157736d106f6" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
W20250626 19:57:31.882036 4609 leader_election.cc:343] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:31.885794 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94327cab01df4338aaec157736d106f6" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:31.887876 4611 leader_election.cc:343] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:31.888931 4611 leader_election.cc:304] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:31.890677 5384 raft_consensus.cc:2747] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:31.891707 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 3bd8dae5699c4725915814044d76a257 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]) successfully deleted
I20250626 19:57:31.897933 5368 raft_consensus.cc:695] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:31.898885 5368 consensus_queue.cc:237] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.911454 4755 leader_election.cc:304] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:31.912549 5360 raft_consensus.cc:2802] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:31.912964 5360 raft_consensus.cc:695] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:31.913767 5360 consensus_queue.cc:237] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:31.949230 5353 ts_tablet_manager.cc:1428] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.261s user 0.007s sys 0.001s
I20250626 19:57:31.954638 5390 tablet.cc:1722] T 7aee4118595346ef990a8a52eb007072 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:31.962373 5388 tablet.cc:1722] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:31.971148 4399 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_5195fbf1909043f8b3fbbe59f976e0c4" } new_table_name: "default.Table_fc5a0fda8b274c989cb3b9189b6a838d" modify_external_catalogs: true
I20250626 19:57:31.986467 5353 tablet_bootstrap.cc:492] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:32.000699 5366 tablet_bootstrap.cc:654] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.013576 5383 raft_consensus.cc:513] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.017035 5353 tablet_bootstrap.cc:654] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.032148 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f1a296c114b4149bc694f6b08a1f2e9" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
W20250626 19:57:32.033572 4609 leader_election.cc:343] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:32.035642 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f1a296c114b4149bc694f6b08a1f2e9" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:32.036144 4978 raft_consensus.cc:3058] T 8f1a296c114b4149bc694f6b08a1f2e9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:32.044091 4978 raft_consensus.cc:2466] T 8f1a296c114b4149bc694f6b08a1f2e9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:32.045312 4611 leader_election.cc:304] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters: 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:32.046154 5368 raft_consensus.cc:2802] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:32.046586 5368 raft_consensus.cc:695] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:32.048183 5368 consensus_queue.cc:237] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.061061 4391 master_service.cc:948] ReplaceTablet: received request to replace tablet 04b0f44c1b564fdf98408881678a90a1 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:32.067195 4394 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_e72dff976c234b9fa5819cb8c6e31d73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:32.067624 5325 ts_tablet_manager.cc:1918] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:32.068567 5366 tablet_bootstrap.cc:492] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:32.069185 5366 ts_tablet_manager.cc:1397] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.202s user 0.009s sys 0.002s
W20250626 19:57:32.069403 4394 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e72dff976c234b9fa5819cb8c6e31d73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:32.072731 5325 log.cc:1199] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/3bd8dae5699c4725915814044d76a257
I20250626 19:57:32.073653 5325 ts_tablet_manager.cc:1939] T 3bd8dae5699c4725915814044d76a257 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:32.075845 5383 leader_election.cc:290] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:32.076822 5366 raft_consensus.cc:357] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.077700 5366 raft_consensus.cc:383] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.078032 5366 raft_consensus.cc:738] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:32.078687 5366 consensus_queue.cc:260] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.083441 5366 ts_tablet_manager.cc:1428] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s
I20250626 19:57:32.085786 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 3bd8dae5699c4725915814044d76a257 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]) successfully deleted
I20250626 19:57:32.091212 5366 tablet_bootstrap.cc:492] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:32.097359 5366 tablet_bootstrap.cc:654] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.120051 5366 tablet_bootstrap.cc:492] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:32.120762 5366 ts_tablet_manager.cc:1397] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.009s
I20250626 19:57:32.122315 5283 raft_consensus.cc:491] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:32.122730 5283 raft_consensus.cc:513] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.125036 5366 raft_consensus.cc:357] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.125856 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3b05549a1434a1fa3d337fdfb96a131" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:32.126123 5366 raft_consensus.cc:383] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.126448 4676 raft_consensus.cc:2466] T a3b05549a1434a1fa3d337fdfb96a131 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:32.126530 5366 raft_consensus.cc:738] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:32.127461 5283 leader_election.cc:290] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:32.127504 5366 consensus_queue.cc:260] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.128474 4912 leader_election.cc:304] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:32.130996 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3b05549a1434a1fa3d337fdfb96a131" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:32.132362 5366 ts_tablet_manager.cc:1428] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
W20250626 19:57:32.133589 4911 leader_election.cc:343] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:32.134243 5366 tablet_bootstrap.cc:492] T a3b05549a1434a1fa3d337fdfb96a131 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:32.133592 5274 raft_consensus.cc:2802] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:32.134678 5274 raft_consensus.cc:491] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:32.135051 5274 raft_consensus.cc:3058] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:32.143538 5366 tablet_bootstrap.cc:654] T a3b05549a1434a1fa3d337fdfb96a131 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.166306 5366 tablet_bootstrap.cc:492] T a3b05549a1434a1fa3d337fdfb96a131 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:32.167366 5366 ts_tablet_manager.cc:1397] T a3b05549a1434a1fa3d337fdfb96a131 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.012s
I20250626 19:57:32.171145 5366 raft_consensus.cc:357] T a3b05549a1434a1fa3d337fdfb96a131 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.174412 5366 raft_consensus.cc:383] T a3b05549a1434a1fa3d337fdfb96a131 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.174978 5366 raft_consensus.cc:738] T a3b05549a1434a1fa3d337fdfb96a131 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:32.176205 5366 consensus_queue.cc:260] T a3b05549a1434a1fa3d337fdfb96a131 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.181454 5366 ts_tablet_manager.cc:1428] T a3b05549a1434a1fa3d337fdfb96a131 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s
I20250626 19:57:32.184394 5366 tablet_bootstrap.cc:492] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:32.194835 5366 tablet_bootstrap.cc:654] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.209338 5353 tablet_bootstrap.cc:492] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:32.211652 5353 ts_tablet_manager.cc:1397] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.225s user 0.036s sys 0.061s
I20250626 19:57:32.215907 5353 raft_consensus.cc:357] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.216698 5353 raft_consensus.cc:383] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.217002 5353 raft_consensus.cc:738] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:32.218250 5353 consensus_queue.cc:260] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.224031 5353 ts_tablet_manager.cc:1428] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20250626 19:57:32.227871 5353 tablet_bootstrap.cc:492] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:32.234498 5283 raft_consensus.cc:491] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:32.234994 5283 raft_consensus.cc:513] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:32.236781 5283 leader_election.cc:290] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:32.239039 4912 leader_election.cc:343] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Not found: Tablet not found: 612f87796cfa44ddbb19b02797031db9
W20250626 19:57:32.240798 4911 leader_election.cc:343] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Not found: Tablet not found: 612f87796cfa44ddbb19b02797031db9
I20250626 19:57:32.241189 5353 tablet_bootstrap.cc:654] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.241233 4911 leader_election.cc:304] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:32.242321 5283 raft_consensus.cc:2747] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:32.193898 5274 raft_consensus.cc:513] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.264302 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3b05549a1434a1fa3d337fdfb96a131" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:32.264947 4676 raft_consensus.cc:3058] T a3b05549a1434a1fa3d337fdfb96a131 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:32.270527 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3b05549a1434a1fa3d337fdfb96a131" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:32.271417 4819 raft_consensus.cc:3058] T a3b05549a1434a1fa3d337fdfb96a131 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:32.275705 4676 raft_consensus.cc:2466] T a3b05549a1434a1fa3d337fdfb96a131 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:32.277204 4912 leader_election.cc:304] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:32.281289 5353 tablet_bootstrap.cc:492] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:32.278050 5283 raft_consensus.cc:2802] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:32.282894 5283 raft_consensus.cc:695] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:32.283715 5283 consensus_queue.cc:237] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.291421 4819 raft_consensus.cc:2466] T a3b05549a1434a1fa3d337fdfb96a131 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:32.281946 5353 ts_tablet_manager.cc:1397] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.015s
I20250626 19:57:32.302212 5353 raft_consensus.cc:357] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.306885 5353 raft_consensus.cc:383] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.307754 5353 raft_consensus.cc:738] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:32.309132 5353 consensus_queue.cc:260] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.316780 5274 leader_election.cc:290] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:32.318164 5353 ts_tablet_manager.cc:1428] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.020s user 0.006s sys 0.002s
I20250626 19:57:32.322695 5353 tablet_bootstrap.cc:492] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
W20250626 19:57:32.326155 4383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.190:33823, user_credentials={real_user=slave}} blocked reactor thread for 68482.7us
W20250626 19:57:32.329103 4383 consensus_peers.cc:487] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af -> Peer e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823): Couldn't send request to peer e6c58006b6be41c1bd6bdb5eb010d360. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250626 19:57:32.346068 5366 tablet_bootstrap.cc:492] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:32.346736 5366 ts_tablet_manager.cc:1397] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.163s user 0.046s sys 0.064s
I20250626 19:57:32.351791 5366 raft_consensus.cc:357] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.352658 5366 raft_consensus.cc:383] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.353071 5366 raft_consensus.cc:738] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:32.354159 5366 consensus_queue.cc:260] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.358706 5366 ts_tablet_manager.cc:1428] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
W20250626 19:57:32.378433 4399 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a31f1a7739544706bff2a500d6026086 has no consensus state
I20250626 19:57:32.379380 4399 master_service.cc:948] ReplaceTablet: received request to replace tablet 4d8ffa9b658c413fa06d02ee6635dd1b from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:32.391682 4954 tablet_service.cc:1468] Processing CreateTablet for tablet 5e20ea7c17c34fbd9c117275e3c14466 (DEFAULT_TABLE table=default.Table_a6836de37d3b41c7aa82c0c1e14ab4b3 [id=089059dab32a41179048a6eb4473a99a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.392302 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 30f1d01851be485f9d766784855ba627 (DEFAULT_TABLE table=default.Table_9db5147373854d58a239e5e5b6048c72 [id=fdc1c1ba57764dac9588fc0f8ad57b75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.393319 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30f1d01851be485f9d766784855ba627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.393491 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e20ea7c17c34fbd9c117275e3c14466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.399819 4798 tablet_service.cc:1468] Processing CreateTablet for tablet 5e20ea7c17c34fbd9c117275e3c14466 (DEFAULT_TABLE table=default.Table_a6836de37d3b41c7aa82c0c1e14ab4b3 [id=089059dab32a41179048a6eb4473a99a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.401582 4798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e20ea7c17c34fbd9c117275e3c14466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.401577 4399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_e5f9396b41eb4c5eb98ec42ec48d7cdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:32.404250 4394 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_d2c793438d234509974deb53c4e61c8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:32.404889 4797 tablet_service.cc:1468] Processing CreateTablet for tablet 30f1d01851be485f9d766784855ba627 (DEFAULT_TABLE table=default.Table_9db5147373854d58a239e5e5b6048c72 [id=fdc1c1ba57764dac9588fc0f8ad57b75]), partition=RANGE (key) PARTITION UNBOUNDED
W20250626 19:57:32.405910 4394 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2c793438d234509974deb53c4e61c8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:32.407459 4652 tablet_service.cc:1468] Processing CreateTablet for tablet 5e20ea7c17c34fbd9c117275e3c14466 (DEFAULT_TABLE table=default.Table_a6836de37d3b41c7aa82c0c1e14ab4b3 [id=089059dab32a41179048a6eb4473a99a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.407742 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 2fda659c30c44c8ba402ac5c7f3598e0 (DEFAULT_TABLE table=default.Table_126ea31ac2c2483e84cedd411a0247b4 [id=ae81e1731bc1461385bf84e443ff0730]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.408533 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e20ea7c17c34fbd9c117275e3c14466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.413739 4797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30f1d01851be485f9d766784855ba627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.414809 4958 tablet_service.cc:1468] Processing CreateTablet for tablet 2fda659c30c44c8ba402ac5c7f3598e0 (DEFAULT_TABLE table=default.Table_126ea31ac2c2483e84cedd411a0247b4 [id=ae81e1731bc1461385bf84e443ff0730]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.415233 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fda659c30c44c8ba402ac5c7f3598e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.415900 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fda659c30c44c8ba402ac5c7f3598e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.418058 4654 tablet_service.cc:1468] Processing CreateTablet for tablet 2fda659c30c44c8ba402ac5c7f3598e0 (DEFAULT_TABLE table=default.Table_126ea31ac2c2483e84cedd411a0247b4 [id=ae81e1731bc1461385bf84e443ff0730]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.419210 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fda659c30c44c8ba402ac5c7f3598e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:32.427349 4399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5f9396b41eb4c5eb98ec42ec48d7cdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:32.430444 4796 tablet_service.cc:1468] Processing CreateTablet for tablet 6b66cda4f8ed410184b434d5083deac4 (DEFAULT_TABLE table=default.Table_fbd8a487fac74f3087145875222b3dab [id=91dc6e79ab2b4c4badbee78be21918fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.432175 4796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b66cda4f8ed410184b434d5083deac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.451918 4957 tablet_service.cc:1468] Processing CreateTablet for tablet 6b66cda4f8ed410184b434d5083deac4 (DEFAULT_TABLE table=default.Table_fbd8a487fac74f3087145875222b3dab [id=91dc6e79ab2b4c4badbee78be21918fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.453088 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b66cda4f8ed410184b434d5083deac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.469132 4655 tablet_service.cc:1468] Processing CreateTablet for tablet 6b66cda4f8ed410184b434d5083deac4 (DEFAULT_TABLE table=default.Table_fbd8a487fac74f3087145875222b3dab [id=91dc6e79ab2b4c4badbee78be21918fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.470279 4655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b66cda4f8ed410184b434d5083deac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.472731 4799 tablet_service.cc:1468] Processing CreateTablet for tablet a31f1a7739544706bff2a500d6026086 (DEFAULT_TABLE table=default.Table_fc5a0fda8b274c989cb3b9189b6a838d [id=ee7713e5a64e4b5b8e32bc5862786d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.473798 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a31f1a7739544706bff2a500d6026086. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.479267 4956 tablet_service.cc:1468] Processing CreateTablet for tablet 30f1d01851be485f9d766784855ba627 (DEFAULT_TABLE table=default.Table_9db5147373854d58a239e5e5b6048c72 [id=fdc1c1ba57764dac9588fc0f8ad57b75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.480578 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30f1d01851be485f9d766784855ba627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.495177 5353 tablet_bootstrap.cc:654] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.507560 4657 tablet_service.cc:1468] Processing CreateTablet for tablet a31f1a7739544706bff2a500d6026086 (DEFAULT_TABLE table=default.Table_fc5a0fda8b274c989cb3b9189b6a838d [id=ee7713e5a64e4b5b8e32bc5862786d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.508721 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a31f1a7739544706bff2a500d6026086. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.519723 5410 tablet_bootstrap.cc:492] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:32.538448 5410 tablet_bootstrap.cc:654] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.551808 5366 tablet_bootstrap.cc:492] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:32.552207 4800 tablet_service.cc:1468] Processing CreateTablet for tablet c990e4eca5cf4a8e8295e9599cf5be29 (DEFAULT_TABLE table=default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.552812 4654 tablet_service.cc:1468] Processing CreateTablet for tablet c990e4eca5cf4a8e8295e9599cf5be29 (DEFAULT_TABLE table=default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.553164 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c990e4eca5cf4a8e8295e9599cf5be29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.553824 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c990e4eca5cf4a8e8295e9599cf5be29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.556537 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:32.569427 5366 tablet_bootstrap.cc:654] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.580523 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:32.590931 5366 tablet_bootstrap.cc:492] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:32.591369 5366 ts_tablet_manager.cc:1397] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.003s
I20250626 19:57:32.596163 5366 raft_consensus.cc:357] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.596835 5366 raft_consensus.cc:383] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.597096 5366 raft_consensus.cc:738] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:32.597728 5366 consensus_queue.cc:260] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.600543 5366 ts_tablet_manager.cc:1428] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20250626 19:57:32.601943 5366 tablet_bootstrap.cc:492] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:32.505872 4959 tablet_service.cc:1468] Processing CreateTablet for tablet a31f1a7739544706bff2a500d6026086 (DEFAULT_TABLE table=default.Table_fc5a0fda8b274c989cb3b9189b6a838d [id=ee7713e5a64e4b5b8e32bc5862786d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.550189 4958 tablet_service.cc:1468] Processing CreateTablet for tablet c990e4eca5cf4a8e8295e9599cf5be29 (DEFAULT_TABLE table=default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:32.613938 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a31f1a7739544706bff2a500d6026086. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.615108 5360 raft_consensus.cc:491] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:32.615682 5360 raft_consensus.cc:513] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:32.617748 5360 leader_election.cc:290] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:32.619911 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f91a78f6d474133976242ea8d259e29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:32.620309 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f91a78f6d474133976242ea8d259e29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:32.621163 4676 raft_consensus.cc:2466] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 2.
I20250626 19:57:32.623435 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c990e4eca5cf4a8e8295e9599cf5be29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:32.629436 5408 raft_consensus.cc:491] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:32.629448 4755 leader_election.cc:304] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:32.629900 5408 raft_consensus.cc:513] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.631793 5360 raft_consensus.cc:2802] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 19:57:32.632320 5360 raft_consensus.cc:491] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:32.632803 5360 raft_consensus.cc:3058] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 2 FOLLOWER]: Advancing to term 3
I20250626 19:57:32.649837 5366 tablet_bootstrap.cc:654] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.652938 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:32.664191 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a2aa555b7a4aad86d923565436f17e" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:32.668010 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a2aa555b7a4aad86d923565436f17e" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:32.668761 4819 raft_consensus.cc:2391] T 76a2aa555b7a4aad86d923565436f17e P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:32.670046 4911 leader_election.cc:304] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:32.670547 5416 raft_consensus.cc:491] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:32.670861 5274 raft_consensus.cc:3058] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:32.670972 5416 raft_consensus.cc:513] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.672154 5410 tablet_bootstrap.cc:492] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:32.678267 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f1a296c114b4149bc694f6b08a1f2e9" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:32.680328 5416 leader_election.cc:290] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:32.687589 4979 raft_consensus.cc:2391] T 8f1a296c114b4149bc694f6b08a1f2e9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93f885c6802341c889a40e9c20fbfff1 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:32.679513 5408 leader_election.cc:290] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:32.682448 4978 raft_consensus.cc:3058] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:32.694904 5410 ts_tablet_manager.cc:1397] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.175s user 0.008s sys 0.006s
I20250626 19:57:32.698279 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f1a296c114b4149bc694f6b08a1f2e9" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:32.700906 4755 leader_election.cc:304] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:32.708206 5416 raft_consensus.cc:3058] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:32.708750 5417 consensus_queue.cc:1035] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:32.711113 5410 raft_consensus.cc:357] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.712085 5410 raft_consensus.cc:383] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.712435 5410 raft_consensus.cc:738] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:32.716264 5410 consensus_queue.cc:260] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.725345 5274 raft_consensus.cc:2747] T 76a2aa555b7a4aad86d923565436f17e P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:32.731709 5410 ts_tablet_manager.cc:1428] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20250626 19:57:32.734356 5366 tablet_bootstrap.cc:492] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:32.735016 5366 ts_tablet_manager.cc:1397] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.133s user 0.011s sys 0.005s
I20250626 19:57:32.737243 5366 raft_consensus.cc:357] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.737937 5366 raft_consensus.cc:383] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.738190 5366 raft_consensus.cc:738] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:32.740782 5410 tablet_bootstrap.cc:492] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:32.752664 5353 tablet_bootstrap.cc:492] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:32.752703 5410 tablet_bootstrap.cc:654] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.753132 5353 ts_tablet_manager.cc:1397] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.433s user 0.010s sys 0.001s
I20250626 19:57:32.745945 5416 raft_consensus.cc:2747] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:32.751945 5366 consensus_queue.cc:260] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.739766 5360 raft_consensus.cc:513] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:32.757445 5366 ts_tablet_manager.cc:1428] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.022s user 0.001s sys 0.004s
I20250626 19:57:32.761101 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f91a78f6d474133976242ea8d259e29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:32.761665 4676 raft_consensus.cc:3058] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 2 FOLLOWER]: Advancing to term 3
I20250626 19:57:32.769935 4676 raft_consensus.cc:2466] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 3.
I20250626 19:57:32.771163 4755 leader_election.cc:304] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:32.772255 5416 raft_consensus.cc:2802] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 19:57:32.772786 5416 raft_consensus.cc:695] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [term 3 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:32.773645 5416 consensus_queue.cc:237] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:32.787134 5366 tablet_bootstrap.cc:492] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:32.795702 5410 tablet_bootstrap.cc:492] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:32.796169 5410 ts_tablet_manager.cc:1397] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.056s user 0.020s sys 0.002s
I20250626 19:57:32.797061 5366 tablet_bootstrap.cc:654] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.798861 5410 raft_consensus.cc:357] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.799543 5410 raft_consensus.cc:383] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.799803 5410 raft_consensus.cc:738] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:32.800428 5410 consensus_queue.cc:260] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.803663 5410 ts_tablet_manager.cc:1428] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250626 19:57:32.808903 5410 tablet_bootstrap.cc:492] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:32.815338 5360 leader_election.cc:290] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 3 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:32.834071 5410 tablet_bootstrap.cc:654] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.833120 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 44960f8db08942879cde6f96e9c620da (table default.Table_27bcd169607a45599dc5a644e6ea5dd7 [id=8df233b9523c49adb3d24727bda24b9e]). Expected version 1 got 0
I20250626 19:57:32.840401 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet a3b05549a1434a1fa3d337fdfb96a131 (table default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]). Expected version 1 got 0
I20250626 19:57:32.843655 5366 tablet_bootstrap.cc:492] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:32.844060 5366 ts_tablet_manager.cc:1397] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.057s user 0.013s sys 0.009s
I20250626 19:57:32.846625 5366 raft_consensus.cc:357] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.847486 5366 raft_consensus.cc:383] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.847816 5366 raft_consensus.cc:738] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:32.848569 5366 consensus_queue.cc:260] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.845839 4395 catalog_manager.cc:5582] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } }
I20250626 19:57:32.859324 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 7aee4118595346ef990a8a52eb007072 (table default.Table_7c1a332875ad49abb1f0f16d7a027691 [id=b5086ce274d44f4dae848455c6274ec8]). Expected version 1 got 0
I20250626 19:57:32.861310 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f91a78f6d474133976242ea8d259e29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:32.861951 5366 ts_tablet_manager.cc:1428] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.018s user 0.006s sys 0.002s
I20250626 19:57:32.862025 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 04b0f44c1b564fdf98408881678a90a1 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:32.862824 5366 tablet_bootstrap.cc:492] T 2fda659c30c44c8ba402ac5c7f3598e0 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:32.866215 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 1cb6bda529624ac5b5a87a03b4a02d56 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:32.867141 5410 tablet_bootstrap.cc:492] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:32.867554 5410 ts_tablet_manager.cc:1397] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.059s user 0.008s sys 0.008s
I20250626 19:57:32.868418 5366 tablet_bootstrap.cc:654] T 2fda659c30c44c8ba402ac5c7f3598e0 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.869931 5410 raft_consensus.cc:357] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.877736 5410 raft_consensus.cc:383] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.878067 5410 raft_consensus.cc:738] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:32.881928 5392 consensus_queue.cc:1035] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:32.884526 5366 tablet_bootstrap.cc:492] T 2fda659c30c44c8ba402ac5c7f3598e0 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:32.884934 5366 ts_tablet_manager.cc:1397] T 2fda659c30c44c8ba402ac5c7f3598e0 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.004s
I20250626 19:57:32.887156 5366 raft_consensus.cc:357] T 2fda659c30c44c8ba402ac5c7f3598e0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.887777 5366 raft_consensus.cc:383] T 2fda659c30c44c8ba402ac5c7f3598e0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.884353 5410 consensus_queue.cc:260] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.888017 5366 raft_consensus.cc:738] T 2fda659c30c44c8ba402ac5c7f3598e0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:32.888561 5366 consensus_queue.cc:260] T 2fda659c30c44c8ba402ac5c7f3598e0 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.891898 5366 ts_tablet_manager.cc:1428] T 2fda659c30c44c8ba402ac5c7f3598e0 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20250626 19:57:32.892717 5366 tablet_bootstrap.cc:492] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:32.902196 5392 consensus_queue.cc:1035] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:32.903993 5366 tablet_bootstrap.cc:654] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.904143 5410 ts_tablet_manager.cc:1428] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.036s user 0.007s sys 0.001s
I20250626 19:57:32.905184 5410 tablet_bootstrap.cc:492] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:32.907923 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 44960f8db08942879cde6f96e9c620da (table default.Table_27bcd169607a45599dc5a644e6ea5dd7 [id=8df233b9523c49adb3d24727bda24b9e]). Expected version 1 got 0
I20250626 19:57:32.917117 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 1cb6bda529624ac5b5a87a03b4a02d56 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:32.921207 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 04b0f44c1b564fdf98408881678a90a1 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:32.922204 5410 tablet_bootstrap.cc:654] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.924111 5416 raft_consensus.cc:491] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:32.925050 5416 raft_consensus.cc:513] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.925788 5366 tablet_bootstrap.cc:492] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:32.926154 5366 ts_tablet_manager.cc:1397] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.001s
I20250626 19:57:32.927007 5416 leader_election.cc:290] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:32.927699 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94327cab01df4338aaec157736d106f6" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:32.928335 4676 raft_consensus.cc:2466] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:32.927883 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 7aee4118595346ef990a8a52eb007072 (table default.Table_7c1a332875ad49abb1f0f16d7a027691 [id=b5086ce274d44f4dae848455c6274ec8]). Expected version 1 got 0
I20250626 19:57:32.928406 5366 raft_consensus.cc:357] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.928810 4974 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94327cab01df4338aaec157736d106f6" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:32.929030 5366 raft_consensus.cc:383] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.929255 5366 raft_consensus.cc:738] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:32.929472 4755 leader_election.cc:304] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:32.929683 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet a3b05549a1434a1fa3d337fdfb96a131 (table default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]). Expected version 1 got 0
I20250626 19:57:32.929802 5366 consensus_queue.cc:260] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.930330 5416 raft_consensus.cc:2802] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:32.930812 5416 raft_consensus.cc:491] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:32.931155 5416 raft_consensus.cc:3058] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:32.935546 5366 ts_tablet_manager.cc:1428] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.009s user 0.002s sys 0.005s
I20250626 19:57:32.936450 5366 tablet_bootstrap.cc:492] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:32.936537 5353 raft_consensus.cc:357] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.937234 5353 raft_consensus.cc:383] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:32.937547 5353 raft_consensus.cc:738] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
W20250626 19:57:32.939266 4756 leader_election.cc:343] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:32.938238 5416 raft_consensus.cc:513] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.941700 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94327cab01df4338aaec157736d106f6" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:32.942245 4676 raft_consensus.cc:3058] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:32.949517 4676 raft_consensus.cc:2466] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:32.950672 4755 leader_election.cc:304] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:32.951536 5360 raft_consensus.cc:2802] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:32.951992 5360 raft_consensus.cc:695] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:32.953356 5360 consensus_queue.cc:237] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.953943 5392 consensus_queue.cc:1035] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:32.958619 5417 raft_consensus.cc:491] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:32.959067 5417 raft_consensus.cc:513] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:32.960836 5417 leader_election.cc:290] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:32.969219 5366 tablet_bootstrap.cc:654] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:32.969774 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04b0f44c1b564fdf98408881678a90a1" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:32.970432 4674 raft_consensus.cc:2466] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:32.972404 5368 raft_consensus.cc:491] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 19:57:32.972991 4383 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet a31f1a7739544706bff2a500d6026086 (table default.Table_fc5a0fda8b274c989cb3b9189b6a838d [id=ee7713e5a64e4b5b8e32bc5862786d05], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a31f1a7739544706bff2a500d6026086 consensus state has no leader
I20250626 19:57:32.973042 5368 raft_consensus.cc:513] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:32.974438 4391 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 04b0f44c1b564fdf98408881678a90a1
I20250626 19:57:32.975890 4391 catalog_manager.cc:6652] ReplaceTablet: tablet 04b0f44c1b564fdf98408881678a90a1 deleted and replaced by tablet 3c03691d021e4ee19d5bb287ab633a3c
I20250626 19:57:32.976266 4391 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34796 (request call id 63) took 1070 ms. Trace:
I20250626 19:57:32.983677 4755 leader_election.cc:304] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:32.985311 5429 raft_consensus.cc:2802] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:32.985670 5429 raft_consensus.cc:491] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:32.985973 5429 raft_consensus.cc:3058] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:32.987814 4974 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04b0f44c1b564fdf98408881678a90a1" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:32.988435 4974 raft_consensus.cc:2466] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:32.990293 5416 leader_election.cc:290] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:32.990993 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet 04b0f44c1b564fdf98408881678a90a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c03691d021e4ee19d5bb287ab633a3c) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:32.976588 4391 rpcz_store.cc:276] 0626 19:57:31.905509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:32.060926 (+155417us) service_pool.cc:225] Handling call
0626 19:57:32.976189 (+915263us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:32.942135 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:32.942607 (+ 472us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:32.942635 (+ 28us) write_op.cc:435] Acquired schema lock
0626 19:57:32.942653 (+ 18us) tablet.cc:660] Decoding operations
0626 19:57:32.943036 (+ 383us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:32.943090 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:32.943116 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:32.943337 (+ 221us) tablet.cc:699] Row locks acquired
0626 19:57:32.943350 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:57:32.943474 (+ 124us) write_op.cc:270] Start()
0626 19:57:32.943563 (+ 89us) write_op.cc:276] Timestamp: P: 1750967852943657 usec, L: 0
0626 19:57:32.943578 (+ 15us) op_driver.cc:348] REPLICATION: starting
0626 19:57:32.944098 (+ 520us) log.cc:844] Serialized 624 byte log entry
0626 19:57:32.969486 (+ 25388us) op_driver.cc:464] REPLICATION: finished
0626 19:57:32.969898 (+ 412us) write_op.cc:301] APPLY: starting
0626 19:57:32.969953 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:32.970053 (+ 100us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:32.970157 (+ 104us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:32.970481 (+ 324us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:32.970675 (+ 194us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:32.970715 (+ 40us) write_op.cc:312] APPLY: finished
0626 19:57:32.970923 (+ 208us) log.cc:844] Serialized 28 byte log entry
0626 19:57:32.971064 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:32.971133 (+ 69us) write_op.cc:454] Released schema lock
0626 19:57:32.971164 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:32.971403 (+ 239us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:32.971431 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":198016,"child_traces":[["op",{"apply.queue_time_us":268,"apply.run_cpu_time_us":1471,"apply.run_wall_time_us":1470,"num_ops":2,"prepare.queue_time_us":128,"prepare.run_cpu_time_us":2331,"prepare.run_wall_time_us":2331,"raft.queue_time_us":3632,"raft.run_cpu_time_us":1018,"raft.run_wall_time_us":1023,"replication_time_us":25783}]]}
I20250626 19:57:32.999778 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet 04b0f44c1b564fdf98408881678a90a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c03691d021e4ee19d5bb287ab633a3c) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:33.002053 5392 consensus_queue.cc:1035] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:33.003634 5368 leader_election.cc:290] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:33.009254 5429 raft_consensus.cc:513] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:33.013131 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e20ea7c17c34fbd9c117275e3c14466" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:33.013756 5433 tablet_replica.cc:331] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:33.013777 4821 raft_consensus.cc:2466] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:33.014748 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04b0f44c1b564fdf98408881678a90a1" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:33.015319 4676 raft_consensus.cc:3058] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:32.938372 5353 consensus_queue.cc:260] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:33.017642 5429 leader_election.cc:290] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:33.025671 4676 raft_consensus.cc:2466] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:33.022225 5416 raft_consensus.cc:491] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:33.026458 5416 raft_consensus.cc:513] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.026816 4755 leader_election.cc:304] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:33.028427 5416 leader_election.cc:290] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:33.028931 5416 raft_consensus.cc:2802] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:33.043712 5366 tablet_bootstrap.cc:492] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:33.044293 5366 ts_tablet_manager.cc:1397] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.108s user 0.010s sys 0.011s
I20250626 19:57:33.045292 5416 raft_consensus.cc:695] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:33.046069 5416 consensus_queue.cc:237] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:33.046833 4974 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e20ea7c17c34fbd9c117275e3c14466" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:33.047070 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b66cda4f8ed410184b434d5083deac4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:33.047560 4676 raft_consensus.cc:2466] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:33.048478 5433 raft_consensus.cc:2241] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:33.048653 4755 leader_election.cc:304] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:33.048965 5433 raft_consensus.cc:2270] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:33.049520 5429 raft_consensus.cc:2802] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:33.049891 5429 raft_consensus.cc:491] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:33.050222 5353 ts_tablet_manager.cc:1428] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.297s user 0.002s sys 0.006s
I20250626 19:57:33.050262 5429 raft_consensus.cc:3058] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:33.053197 4797 tablet_service.cc:1515] Processing DeleteTablet for tablet 04b0f44c1b564fdf98408881678a90a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c03691d021e4ee19d5bb287ab633a3c) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:33.056428 5366 raft_consensus.cc:357] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.057276 5366 raft_consensus.cc:383] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:33.057600 5366 raft_consensus.cc:738] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:33.065496 5433 ts_tablet_manager.cc:1905] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:33.067334 4974 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94327cab01df4338aaec157736d106f6" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:33.068217 4974 raft_consensus.cc:3058] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:33.070017 5366 consensus_queue.cc:260] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.072129 5366 ts_tablet_manager.cc:1428] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.027s user 0.004s sys 0.002s
I20250626 19:57:33.072556 5410 tablet_bootstrap.cc:492] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:33.073005 5410 ts_tablet_manager.cc:1397] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.168s user 0.002s sys 0.007s
I20250626 19:57:33.066340 5429 raft_consensus.cc:513] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.076366 4974 raft_consensus.cc:2466] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:33.077736 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b66cda4f8ed410184b434d5083deac4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:33.078347 4676 raft_consensus.cc:3058] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:33.076071 5410 raft_consensus.cc:357] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.085758 5410 raft_consensus.cc:383] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:33.086037 5410 raft_consensus.cc:738] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:33.086736 5410 consensus_queue.cc:260] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.087895 5429 leader_election.cc:290] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:33.089478 5410 ts_tablet_manager.cc:1428] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.016s user 0.004s sys 0.004s
I20250626 19:57:33.092954 4676 raft_consensus.cc:2466] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:33.093549 5410 tablet_bootstrap.cc:492] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:33.096182 4755 leader_election.cc:304] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
W20250626 19:57:33.096994 4611 leader_election.cc:343] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:33.097074 5429 raft_consensus.cc:2802] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:33.097543 5429 raft_consensus.cc:695] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:33.098588 5429 consensus_queue.cc:237] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.116977 5410 tablet_bootstrap.cc:654] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:33.126906 4657 tablet_service.cc:1468] Processing CreateTablet for tablet c92deaa237064dee8f866e3a7a355ca4 (DEFAULT_TABLE table=default.Table_a2a4a2c9c59248dfb73f7db218b0d982 [id=0d00dcf2b9f64037b166b83f38e1a72a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.127674 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 44960f8db08942879cde6f96e9c620da (table default.Table_27bcd169607a45599dc5a644e6ea5dd7 [id=8df233b9523c49adb3d24727bda24b9e]). Expected version 1 got 0
I20250626 19:57:33.127928 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c92deaa237064dee8f866e3a7a355ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.136755 5438 tablet_replica.cc:331] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:33.137735 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 1cb6bda529624ac5b5a87a03b4a02d56 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:33.137874 5438 raft_consensus.cc:2241] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:33.138998 5438 pending_rounds.cc:70] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1: Trying to abort 1 pending ops.
I20250626 19:57:33.145223 4654 tablet_service.cc:1468] Processing CreateTablet for tablet f43aed8f07844c5ca727ae6df63896df (DEFAULT_TABLE table=default.Table_d6eed56e9d514833bfe29ff3aba9718a [id=07c980b0511a43c0aa406597788cd05b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.146209 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43aed8f07844c5ca727ae6df63896df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.163575 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 33a75d223c274c049cf25f79fcc4b961 (DEFAULT_TABLE table=default.Table_d0820205b84340099fad74c11e7ab51c [id=f2095000e7e14f989056caadebfec0c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.164538 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a75d223c274c049cf25f79fcc4b961. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.141993 4399 catalog_manager.cc:5582] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:33.188673 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet a3b05549a1434a1fa3d337fdfb96a131 (table default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]). Expected version 1 got 0
I20250626 19:57:33.189554 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet a31f1a7739544706bff2a500d6026086 (table default.Table_fc5a0fda8b274c989cb3b9189b6a838d [id=ee7713e5a64e4b5b8e32bc5862786d05]). Expected version 1 got 0
I20250626 19:57:33.193325 4609 leader_election.cc:304] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:33.095958 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04b0f44c1b564fdf98408881678a90a1" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:33.115245 5274 raft_consensus.cc:491] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93f885c6802341c889a40e9c20fbfff1)
I20250626 19:57:33.199112 5274 raft_consensus.cc:513] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:33.122968 5353 tablet_bootstrap.cc:492] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:33.123805 4957 tablet_service.cc:1468] Processing CreateTablet for tablet c92deaa237064dee8f866e3a7a355ca4 (DEFAULT_TABLE table=default.Table_a2a4a2c9c59248dfb73f7db218b0d982 [id=0d00dcf2b9f64037b166b83f38e1a72a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.201052 5274 leader_election.cc:290] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:33.126252 5431 tablet_replica.cc:331] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:33.209213 4977 raft_consensus.cc:3058] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:33.213270 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c92deaa237064dee8f866e3a7a355ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.168602 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b66cda4f8ed410184b434d5083deac4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:33.174387 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b66cda4f8ed410184b434d5083deac4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:33.240404 4394 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 44960f8db08942879cde6f96e9c620da (table default.Table_27bcd169607a45599dc5a644e6ea5dd7 [id=8df233b9523c49adb3d24727bda24b9e]). Expected version 1 got 0
I20250626 19:57:33.241400 4394 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet a3b05549a1434a1fa3d337fdfb96a131 (table default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]). Expected version 1 got 0
I20250626 19:57:33.162492 4798 tablet_service.cc:1468] Processing CreateTablet for tablet f43aed8f07844c5ca727ae6df63896df (DEFAULT_TABLE table=default.Table_d6eed56e9d514833bfe29ff3aba9718a [id=07c980b0511a43c0aa406597788cd05b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.163929 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 33a75d223c274c049cf25f79fcc4b961 (DEFAULT_TABLE table=default.Table_d0820205b84340099fad74c11e7ab51c [id=f2095000e7e14f989056caadebfec0c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.245052 5438 pending_rounds.cc:77] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964326085419008 op_type: NO_OP noop_request { }
I20250626 19:57:33.245388 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a75d223c274c049cf25f79fcc4b961. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.245568 5438 raft_consensus.cc:2887] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:57:33.245905 5438 raft_consensus.cc:2270] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:33.247411 4394 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 1cb6bda529624ac5b5a87a03b4a02d56 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:33.249399 4394 catalog_manager.cc:5582] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:33.251276 4798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43aed8f07844c5ca727ae6df63896df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.261045 5433 ts_tablet_manager.cc:1918] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:33.261595 5433 log.cc:1199] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/04b0f44c1b564fdf98408881678a90a1
I20250626 19:57:33.164172 4797 tablet_service.cc:1468] Processing CreateTablet for tablet c92deaa237064dee8f866e3a7a355ca4 (DEFAULT_TABLE table=default.Table_a2a4a2c9c59248dfb73f7db218b0d982 [id=0d00dcf2b9f64037b166b83f38e1a72a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250626 19:57:33.172881 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 54444us
I20250626 19:57:33.274055 4797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c92deaa237064dee8f866e3a7a355ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.273362 4394 catalog_manager.cc:5582] T 76a2aa555b7a4aad86d923565436f17e P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:33.280306 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 61fcd55f6d7b4a369383d4f73d5f483d (DEFAULT_TABLE table=default.Table_427be70c41324505a21930cf7e51915f [id=8275f0cb6d6741adad6da0d05b8eca82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.281421 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61fcd55f6d7b4a369383d4f73d5f483d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.282517 5433 ts_tablet_manager.cc:1939] T 04b0f44c1b564fdf98408881678a90a1 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:33.283890 4977 raft_consensus.cc:2466] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:33.284730 5410 tablet_bootstrap.cc:492] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:33.285138 5410 ts_tablet_manager.cc:1397] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.192s user 0.011s sys 0.001s
I20250626 19:57:33.285354 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 7facb77eb579448f9533d285cba8650e (DEFAULT_TABLE table=default.Table_e72dff976c234b9fa5819cb8c6e31d73 [id=e4b13e1cb6c143339891b9deda4ee7bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.286394 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7facb77eb579448f9533d285cba8650e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.287478 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 61fcd55f6d7b4a369383d4f73d5f483d (DEFAULT_TABLE table=default.Table_427be70c41324505a21930cf7e51915f [id=8275f0cb6d6741adad6da0d05b8eca82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.292469 4652 tablet_service.cc:1468] Processing CreateTablet for tablet fd6543699e254400b3516aece4486fe0 (DEFAULT_TABLE table=default.Table_fde5b6bd4ee8417e85ffccad3bd2a581 [id=ab0bf254569748ba8d31931d4bfb105d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.293397 4654 tablet_service.cc:1468] Processing CreateTablet for tablet 7facb77eb579448f9533d285cba8650e (DEFAULT_TABLE table=default.Table_e72dff976c234b9fa5819cb8c6e31d73 [id=e4b13e1cb6c143339891b9deda4ee7bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.294435 4657 tablet_service.cc:1468] Processing CreateTablet for tablet 4c0338c05a974e0f96865398722c5e8f (DEFAULT_TABLE table=default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.296028 4796 tablet_service.cc:1468] Processing CreateTablet for tablet 4c0338c05a974e0f96865398722c5e8f (DEFAULT_TABLE table=default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.297083 4796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0338c05a974e0f96865398722c5e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.299178 5438 ts_tablet_manager.cc:1905] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:33.304522 4795 tablet_service.cc:1468] Processing CreateTablet for tablet fd6543699e254400b3516aece4486fe0 (DEFAULT_TABLE table=default.Table_fde5b6bd4ee8417e85ffccad3bd2a581 [id=ab0bf254569748ba8d31931d4bfb105d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.306444 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd6543699e254400b3516aece4486fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.306491 5444 raft_consensus.cc:2802] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:33.306877 5444 raft_consensus.cc:491] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:33.307179 5444 raft_consensus.cc:3058] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:33.281723 5368 raft_consensus.cc:491] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:33.311058 5368 raft_consensus.cc:513] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.335284 5366 tablet_bootstrap.cc:492] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:33.340756 5366 tablet_bootstrap.cc:654] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:33.371969 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7facb77eb579448f9533d285cba8650e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.372864 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0338c05a974e0f96865398722c5e8f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:33.377641 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 184543us
I20250626 19:57:33.378012 5416 consensus_queue.cc:1035] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:33.366966 5368 leader_election.cc:290] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:33.371892 5444 raft_consensus.cc:513] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.381827 5410 raft_consensus.cc:357] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.382259 5444 leader_election.cc:290] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:33.385319 5366 tablet_bootstrap.cc:492] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:33.385736 5366 ts_tablet_manager.cc:1397] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.004s
I20250626 19:57:33.386287 5438 ts_tablet_manager.cc:1918] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:33.393061 5438 log.cc:1199] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/04b0f44c1b564fdf98408881678a90a1
I20250626 19:57:33.399570 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd6543699e254400b3516aece4486fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.402316 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61fcd55f6d7b4a369383d4f73d5f483d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.409560 4396 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 44960f8db08942879cde6f96e9c620da (table default.Table_27bcd169607a45599dc5a644e6ea5dd7 [id=8df233b9523c49adb3d24727bda24b9e]). Expected version 1 got 0
I20250626 19:57:33.410459 4396 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet a3b05549a1434a1fa3d337fdfb96a131 (table default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]). Expected version 1 got 0
I20250626 19:57:33.411499 4396 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 1cb6bda529624ac5b5a87a03b4a02d56 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:33.413240 4396 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 7aee4118595346ef990a8a52eb007072 (table default.Table_7c1a332875ad49abb1f0f16d7a027691 [id=b5086ce274d44f4dae848455c6274ec8]). Expected version 1 got 0
I20250626 19:57:33.396793 5438 ts_tablet_manager.cc:1939] T 04b0f44c1b564fdf98408881678a90a1 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:33.388563 5366 raft_consensus.cc:357] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:33.419654 5366 raft_consensus.cc:383] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:33.422267 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 7391b9eaea3e426ba4118fc8b53b72a6 (DEFAULT_TABLE table=default.Table_e5f9396b41eb4c5eb98ec42ec48d7cdc [id=bad1d85ef3a347ab9abe78289306ca4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.429749 5366 raft_consensus.cc:738] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:33.437492 4391 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 44960f8db08942879cde6f96e9c620da (table default.Table_27bcd169607a45599dc5a644e6ea5dd7 [id=8df233b9523c49adb3d24727bda24b9e]). Expected version 1 got 0
I20250626 19:57:33.438558 4391 catalog_manager.cc:5582] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:33.430392 5366 consensus_queue.cc:260] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:33.382525 5410 raft_consensus.cc:383] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:33.443009 5410 raft_consensus.cc:738] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:33.445034 4391 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet a3b05549a1434a1fa3d337fdfb96a131 (table default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]). Expected version 1 got 0
I20250626 19:57:33.446332 4391 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 1cb6bda529624ac5b5a87a03b4a02d56 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:33.448207 5416 raft_consensus.cc:491] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:33.448726 5416 raft_consensus.cc:513] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.452009 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 65af5ae255d74143976392bccc50f3ba (DEFAULT_TABLE table=default.Table_d2c793438d234509974deb53c4e61c8b [id=84b3a43b3e5b42849f8b343a97906309]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.452942 5416 leader_election.cc:290] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:33.454285 5366 ts_tablet_manager.cc:1428] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.068s user 0.008s sys 0.000s
I20250626 19:57:33.463835 4655 tablet_service.cc:1468] Processing CreateTablet for tablet 7391b9eaea3e426ba4118fc8b53b72a6 (DEFAULT_TABLE table=default.Table_e5f9396b41eb4c5eb98ec42ec48d7cdc [id=bad1d85ef3a347ab9abe78289306ca4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.465232 4655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7391b9eaea3e426ba4118fc8b53b72a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.466835 4798 tablet_service.cc:1468] Processing CreateTablet for tablet 3c03691d021e4ee19d5bb287ab633a3c (DEFAULT_TABLE table=default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.475975 4397 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 1e453a4f8ca44f4a940928d9f4fcd5be
I20250626 19:57:33.477476 4397 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34796 (request call id 59) took 1805 ms. Trace:
I20250626 19:57:33.477749 4397 rpcz_store.cc:276] 0626 19:57:31.672001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:31.672196 (+ 195us) service_pool.cc:225] Handling call
0626 19:57:31.697258 (+ 25062us) spinlock_profiling.cc:243] Waited 24.4 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe791557046 00007fe78f2781a4 00007fe78f278132 00007fe78f278061 00007fe78f277fdc 00007fe78f277292 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:31.763533 (+ 66275us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:31.764075 (+ 542us) catalog_manager.cc:2790] Modifying in-memory table state
0626 19:57:31.764201 (+ 126us) catalog_manager.cc:2799] Locking tablets
0626 19:57:32.387219 (+623018us) catalog_manager.cc:2813] Removing table and tablets from system table
0626 19:57:32.780652 (+393433us) catalog_manager.cc:2831] Removing table from by-name map
0626 19:57:32.781235 (+ 583us) catalog_manager.cc:2847] Committing in-memory state
0626 19:57:33.475779 (+694544us) catalog_manager.cc:2851] Aborting table tasks
0626 19:57:33.477394 (+ 1615us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:32.388962 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:32.389429 (+ 467us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:32.389458 (+ 29us) write_op.cc:435] Acquired schema lock
0626 19:57:32.389473 (+ 15us) tablet.cc:660] Decoding operations
0626 19:57:32.389816 (+ 343us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:32.389856 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:32.389884 (+ 28us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:32.390149 (+ 265us) tablet.cc:699] Row locks acquired
0626 19:57:32.390166 (+ 17us) write_op.cc:260] PREPARE: finished
0626 19:57:32.390300 (+ 134us) write_op.cc:270] Start()
0626 19:57:32.390396 (+ 96us) write_op.cc:276] Timestamp: P: 1750967852390265 usec, L: 0
0626 19:57:32.390417 (+ 21us) op_driver.cc:348] REPLICATION: starting
0626 19:57:32.411493 (+ 21076us) log.cc:844] Serialized 761 byte log entry
0626 19:57:32.412113 (+ 620us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72b481 00007fe78f7a2a18 00007fe78fc0f811 00007fe78fc0efe8 00007fe78fc13c81 00007fe78fc13c39 00007fe78fc13bc9 00007fe78fc13b91 00007fe78fc12dbd 000000000059bcd4 000000000059bb09
0626 19:57:32.772351 (+360238us) op_driver.cc:464] REPLICATION: finished
0626 19:57:32.774779 (+ 2428us) write_op.cc:301] APPLY: starting
0626 19:57:32.774878 (+ 99us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:32.775082 (+ 204us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:32.775132 (+ 50us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:32.775578 (+ 446us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:32.775849 (+ 271us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:32.775898 (+ 49us) write_op.cc:312] APPLY: finished
0626 19:57:32.776214 (+ 316us) log.cc:844] Serialized 28 byte log entry
0626 19:57:32.776423 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:32.776509 (+ 86us) write_op.cc:454] Released schema lock
0626 19:57:32.776559 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:32.777012 (+ 453us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:32.777039 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":56080128,"child_traces":[["op",{"apply.queue_time_us":2130,"apply.run_cpu_time_us":2224,"apply.run_wall_time_us":2223,"num_ops":2,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":3281,"prepare.run_wall_time_us":23374,"raft.queue_time_us":313,"raft.run_cpu_time_us":16,"raft.run_wall_time_us":17,"replication_time_us":381604,"spinlock_wait_cycles":46811008}]]}
I20250626 19:57:33.484978 4794 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e453a4f8ca44f4a940928d9f4fcd5be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:31 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:33.485419 4397 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_7a020d26ba3141acaca632ae611a203b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:33.487056 4651 tablet_service.cc:1468] Processing CreateTablet for tablet 3c03691d021e4ee19d5bb287ab633a3c (DEFAULT_TABLE table=default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]), partition=RANGE (key) PARTITION UNBOUNDED
W20250626 19:57:33.487336 4397 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a020d26ba3141acaca632ae611a203b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:33.483603 5410 consensus_queue.cc:260] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.491047 4392 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 4d8ffa9b658c413fa06d02ee6635dd1b
I20250626 19:57:33.492573 4392 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34796 (request call id 57) took 1914 ms. Trace:
I20250626 19:57:33.488188 4651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c03691d021e4ee19d5bb287ab633a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.492915 4392 rpcz_store.cc:276] 0626 19:57:31.577795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:31.578097 (+ 302us) service_pool.cc:225] Handling call
0626 19:57:31.595725 (+ 17628us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe791557046 00007fe78f2781a4 00007fe78f278132 00007fe78f278061 00007fe78f277fdc 00007fe78f277292 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:31.597016 (+ 1291us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:31.597712 (+ 696us) catalog_manager.cc:2790] Modifying in-memory table state
0626 19:57:31.597890 (+ 178us) catalog_manager.cc:2799] Locking tablets
0626 19:57:31.598608 (+ 718us) catalog_manager.cc:2813] Removing table and tablets from system table
0626 19:57:31.965403 (+366795us) catalog_manager.cc:2831] Removing table from by-name map
0626 19:57:31.965627 (+ 224us) catalog_manager.cc:2847] Committing in-memory state
0626 19:57:33.490839 (+1525212us) catalog_manager.cc:2851] Aborting table tasks
0626 19:57:33.492516 (+ 1677us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:31.699650 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:31.724024 (+ 24374us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:31.724385 (+ 361us) write_op.cc:435] Acquired schema lock
0626 19:57:31.724659 (+ 274us) tablet.cc:660] Decoding operations
0626 19:57:31.736523 (+ 11864us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:31.737576 (+ 1053us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:31.738357 (+ 781us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:31.806771 (+ 68414us) tablet.cc:699] Row locks acquired
0626 19:57:31.806790 (+ 19us) write_op.cc:260] PREPARE: finished
0626 19:57:31.806918 (+ 128us) write_op.cc:270] Start()
0626 19:57:31.807009 (+ 91us) write_op.cc:276] Timestamp: P: 1750967851807053 usec, L: 0
0626 19:57:31.807027 (+ 18us) op_driver.cc:348] REPLICATION: starting
0626 19:57:31.807527 (+ 500us) log.cc:844] Serialized 758 byte log entry
0626 19:57:31.960626 (+153099us) op_driver.cc:464] REPLICATION: finished
0626 19:57:31.960918 (+ 292us) write_op.cc:301] APPLY: starting
0626 19:57:31.960989 (+ 71us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:31.961114 (+ 125us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:31.961216 (+ 102us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:31.961445 (+ 229us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:31.961608 (+ 163us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:31.961656 (+ 48us) write_op.cc:312] APPLY: finished
0626 19:57:31.961841 (+ 185us) log.cc:844] Serialized 28 byte log entry
0626 19:57:31.962003 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:31.962067 (+ 64us) write_op.cc:454] Released schema lock
0626 19:57:31.962096 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:31.962326 (+ 230us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:31.962349 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":38079872,"child_traces":[["op",{"apply.queue_time_us":90,"apply.run_cpu_time_us":1398,"apply.run_wall_time_us":1398,"num_ops":2,"prepare.queue_time_us":72690,"prepare.run_cpu_time_us":3456,"prepare.run_wall_time_us":108407,"raft.queue_time_us":1084,"raft.run_cpu_time_us":17,"raft.run_wall_time_us":18,"replication_time_us":153462,"spinlock_wait_cycles":9216}]]}
I20250626 19:57:33.498915 4392 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_a6836de37d3b41c7aa82c0c1e14ab4b3" } new_table_name: "default.Table_84725c81a9a9432c8232f87fb7627f6a" modify_external_catalogs: true
I20250626 19:57:33.492545 4650 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e453a4f8ca44f4a940928d9f4fcd5be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:31 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:33.483603 4653 tablet_service.cc:1468] Processing CreateTablet for tablet 65af5ae255d74143976392bccc50f3ba (DEFAULT_TABLE table=default.Table_d2c793438d234509974deb53c4e61c8b [id=84b3a43b3e5b42849f8b343a97906309]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.501055 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30f1d01851be485f9d766784855ba627" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:33.501315 4794 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d8ffa9b658c413fa06d02ee6635dd1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:31 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:33.501729 4674 raft_consensus.cc:2466] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:33.503171 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65af5ae255d74143976392bccc50f3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.503154 5416 raft_consensus.cc:491] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:33.503585 5416 raft_consensus.cc:513] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.505520 4755 leader_election.cc:304] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:33.505771 5416 leader_election.cc:290] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:33.506565 5416 raft_consensus.cc:2802] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:33.506886 5416 raft_consensus.cc:491] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:33.506973 4393 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34796 (request call id 60) took 1833 ms. Trace:
I20250626 19:57:33.507175 5416 raft_consensus.cc:3058] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:33.507784 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a31f1a7739544706bff2a500d6026086" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:33.509971 4755 leader_election.cc:343] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:33.507418 4393 rpcz_store.cc:276] 0626 19:57:31.672968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:31.766412 (+ 93444us) service_pool.cc:225] Handling call
0626 19:57:31.767397 (+ 985us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:31.768996 (+ 1599us) catalog_manager.cc:3683] Apply alter schema
0626 19:57:31.769839 (+ 843us) catalog_manager.cc:3715] Acquired catalog manager lock
0626 19:57:31.770029 (+ 190us) catalog_manager.cc:3873] Updating metadata on disk
0626 19:57:32.376648 (+606619us) catalog_manager.cc:3904] Committing alterations to in-memory state
0626 19:57:33.506723 (+1130075us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:31.970533 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:32.057936 (+ 87403us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:32.057965 (+ 29us) write_op.cc:435] Acquired schema lock
0626 19:57:32.057980 (+ 15us) tablet.cc:660] Decoding operations
0626 19:57:32.058270 (+ 290us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:32.058305 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:32.058337 (+ 32us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:32.058481 (+ 144us) tablet.cc:699] Row locks acquired
0626 19:57:32.058508 (+ 27us) write_op.cc:260] PREPARE: finished
0626 19:57:32.058627 (+ 119us) write_op.cc:270] Start()
0626 19:57:32.058711 (+ 84us) write_op.cc:276] Timestamp: P: 1750967852058669 usec, L: 0
0626 19:57:32.058730 (+ 19us) op_driver.cc:348] REPLICATION: starting
0626 19:57:32.059287 (+ 557us) log.cc:844] Serialized 404 byte log entry
0626 19:57:32.374498 (+315211us) op_driver.cc:464] REPLICATION: finished
0626 19:57:32.375469 (+ 971us) write_op.cc:301] APPLY: starting
0626 19:57:32.375528 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:32.375615 (+ 87us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:32.375705 (+ 90us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:32.375852 (+ 147us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:32.375979 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:32.376038 (+ 59us) write_op.cc:312] APPLY: finished
0626 19:57:32.376207 (+ 169us) log.cc:844] Serialized 22 byte log entry
0626 19:57:32.376357 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:32.376425 (+ 68us) write_op.cc:454] Released schema lock
0626 19:57:32.376455 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:32.376681 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:32.376705 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":818,"apply.run_cpu_time_us":1184,"apply.run_wall_time_us":1184,"num_ops":1,"prepare.queue_time_us":199269,"prepare.run_cpu_time_us":2185,"prepare.run_wall_time_us":89051,"raft.queue_time_us":2330,"raft.run_cpu_time_us":16,"raft.run_wall_time_us":17,"replication_time_us":315666}]]}
I20250626 19:57:33.512902 4393 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_c580e8a6ee5d417caa2f7d6811a68349"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:33.513968 4650 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d8ffa9b658c413fa06d02ee6635dd1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:31 UTC) from {username='slave'} at 127.0.0.1:49550
W20250626 19:57:33.515646 4393 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c580e8a6ee5d417caa2f7d6811a68349 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:33.521391 4398 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9] alter complete (version 1)
I20250626 19:57:33.528695 5433 tablet_replica.cc:331] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:33.529620 5433 raft_consensus.cc:2241] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:33.530150 5433 raft_consensus.cc:2270] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:33.532213 5433 ts_tablet_manager.cc:1905] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:33.533612 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 04b0f44c1b564fdf98408881678a90a1 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]) successfully deleted
W20250626 19:57:33.534291 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 150227us
I20250626 19:57:33.541426 5410 ts_tablet_manager.cc:1428] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.256s user 0.011s sys 0.009s
I20250626 19:57:33.162496 4952 tablet_service.cc:1468] Processing CreateTablet for tablet 61fcd55f6d7b4a369383d4f73d5f483d (DEFAULT_TABLE table=default.Table_427be70c41324505a21930cf7e51915f [id=8275f0cb6d6741adad6da0d05b8eca82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.579622 4952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61fcd55f6d7b4a369383d4f73d5f483d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.158793 4953 tablet_service.cc:1468] Processing CreateTablet for tablet 33a75d223c274c049cf25f79fcc4b961 (DEFAULT_TABLE table=default.Table_d0820205b84340099fad74c11e7ab51c [id=f2095000e7e14f989056caadebfec0c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.584676 4953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a75d223c274c049cf25f79fcc4b961. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.137697 4954 tablet_service.cc:1468] Processing CreateTablet for tablet f43aed8f07844c5ca727ae6df63896df (DEFAULT_TABLE table=default.Table_d6eed56e9d514833bfe29ff3aba9718a [id=07c980b0511a43c0aa406597788cd05b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.577804 4959 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41356 (request call id 15) took 1072 ms. Trace:
I20250626 19:57:33.301340 5353 tablet_bootstrap.cc:654] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:33.607949 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:33.613229 4399 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: default.Table_7c1a332875ad49abb1f0f16d7a027691 [id=b5086ce274d44f4dae848455c6274ec8] alter complete (version 1)
I20250626 19:57:33.613617 5416 raft_consensus.cc:513] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
W20250626 19:57:33.577131 4958 consensus_meta.cc:287] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent flushing consensus metadata: real 0.663s user 0.012s sys 0.000s
I20250626 19:57:33.592134 5431 raft_consensus.cc:2241] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:33.604660 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43aed8f07844c5ca727ae6df63896df. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:33.605993 4956 consensus_meta.cc:287] T 30f1d01851be485f9d766784855ba627 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent flushing consensus metadata: real 0.765s user 0.006s sys 0.005s
I20250626 19:57:33.616268 5416 leader_election.cc:290] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:33.616407 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:33.618937 5444 raft_consensus.cc:491] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:33.619503 5444 raft_consensus.cc:513] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.626159 5431 raft_consensus.cc:2270] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:33.628165 4674 raft_consensus.cc:1273] T 9f91a78f6d474133976242ea8d259e29 P 4d3be519a0044a2f921a503412f72e86 [term 3 FOLLOWER]: Refusing update from remote peer 93f885c6802341c889a40e9c20fbfff1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250626 19:57:33.629097 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30f1d01851be485f9d766784855ba627" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:33.629648 4677 raft_consensus.cc:3058] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:33.626807 5410 tablet_bootstrap.cc:492] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:33.621428 5444 leader_election.cc:290] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:33.640579 5431 ts_tablet_manager.cc:1905] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:33.643524 5416 consensus_queue.cc:1035] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:33.589164 4959 rpcz_store.cc:276] 0626 19:57:32.505169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:32.505306 (+ 137us) service_pool.cc:225] Handling call
0626 19:57:32.613424 (+108118us) spinlock_profiling.cc:243] Waited 5.86 ms on lock 0x7b5800000098. stack: 00007f2c08de0483 00007f2c08de0380 00007f2c0895c574 00000000004fa174 00000000004fa109 00007f2c08c559a1 00007f2c08dd4eb1 00007f2c08dd4d62 00007f2c11a498f9 00007f2c10cd1ac0 00007f2c10db4e71 00007f2c10d92203 00007f2c10d42c99 00007f2c0f411d34 00007f2c0f411cc2 00007f2c0f411bf1
0626 19:57:32.613531 (+ 107us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0626 19:57:32.613603 (+ 72us) ts_tablet_manager.cc:676] Creating new metadata...
0626 19:57:32.800787 (+187184us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:33.190171 (+389384us) tablet_replica.cc:206] Creating consensus instance
0626 19:57:33.190735 (+ 564us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13488000}
I20250626 19:57:33.647680 4397 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 67) took 1319 ms. Trace:
I20250626 19:57:33.648727 4958 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41356 (request call id 16) took 1099 ms. Trace:
I20250626 19:57:33.649096 4958 rpcz_store.cc:276] 0626 19:57:32.549242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:32.549504 (+ 262us) service_pool.cc:225] Handling call
0626 19:57:32.622861 (+ 73357us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b5800000098. stack: 00007f2c08de0483 00007f2c08de0380 00007f2c0895c574 00000000004fa174 00000000004fa109 00007f2c08c559a1 00007f2c08dd4eb1 00007f2c08dd4d62 00007f2c11a498f9 00007f2c10cd1ac0 00007f2c10db4e71 00007f2c10d92203 00007f2c10d42c99 00007f2c0f411d34 00007f2c0f411cc2 00007f2c0f411bf1
0626 19:57:32.622977 (+ 116us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0626 19:57:32.623057 (+ 80us) ts_tablet_manager.cc:676] Creating new metadata...
0626 19:57:32.913311 (+290254us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:33.648121 (+734810us) tablet_replica.cc:206] Creating consensus instance
0626 19:57:33.648661 (+ 540us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36269440}
I20250626 19:57:33.650900 4956 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41356 (request call id 11) took 1260 ms. Trace:
I20250626 19:57:33.651271 4956 rpcz_store.cc:276] 0626 19:57:32.390122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:32.391055 (+ 933us) service_pool.cc:225] Handling call
0626 19:57:32.480042 (+ 88987us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0626 19:57:32.480144 (+ 102us) ts_tablet_manager.cc:676] Creating new metadata...
0626 19:57:32.714435 (+234291us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:33.650418 (+935983us) tablet_replica.cc:206] Creating consensus instance
0626 19:57:33.650847 (+ 429us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 19:57:33.659883 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e20ea7c17c34fbd9c117275e3c14466" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:33.660452 4821 raft_consensus.cc:3058] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:33.663631 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30f1d01851be485f9d766784855ba627" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:33.664546 4820 raft_consensus.cc:2391] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 93f885c6802341c889a40e9c20fbfff1 in this term.
I20250626 19:57:33.671633 4794 tablet_service.cc:1515] Processing DeleteTablet for tablet ea7374ad6ba64e46b7df9852fa33f145 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:33.671875 4798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c03691d021e4ee19d5bb287ab633a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.670549 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f91a78f6d474133976242ea8d259e29" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:33.675751 4393 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 69) took 1068 ms. Trace:
I20250626 19:57:33.686923 5438 tablet_replica.cc:331] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:33.687081 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 04b0f44c1b564fdf98408881678a90a1 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]) successfully deleted
I20250626 19:57:33.688864 5438 raft_consensus.cc:2241] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:33.700275 4956 tablet_service.cc:1468] Processing CreateTablet for tablet 4c0338c05a974e0f96865398722c5e8f (DEFAULT_TABLE table=default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.701529 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0338c05a974e0f96865398722c5e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.703671 5438 raft_consensus.cc:2270] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:33.705106 4797 tablet_service.cc:1515] Processing DeleteTablet for tablet afd1fafb53434957b002fad5c77431d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:33.705667 4794 tablet_service.cc:1515] Processing DeleteTablet for tablet 512dc911bfce49b29c6fc8cd90523723 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:33.705780 4796 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a8f494f8614e7ebb8ca880b6c80e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:33.706406 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 5785508e37624f02bc2f6cc93007e3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:28 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:33.689514 5366 tablet_bootstrap.cc:492] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:33.689693 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7391b9eaea3e426ba4118fc8b53b72a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.706737 4677 raft_consensus.cc:2466] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:33.708561 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65af5ae255d74143976392bccc50f3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.707898 4755 leader_election.cc:304] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:33.710013 5429 raft_consensus.cc:2802] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:33.710417 5429 raft_consensus.cc:695] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:33.711139 5429 consensus_queue.cc:237] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.712813 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bd8dae5699c4725915814044d76a257 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c0338c05a974e0f96865398722c5e8f) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:33.711154 4958 tablet_service.cc:1468] Processing CreateTablet for tablet fd6543699e254400b3516aece4486fe0 (DEFAULT_TABLE table=default.Table_fde5b6bd4ee8417e85ffccad3bd2a581 [id=ab0bf254569748ba8d31931d4bfb105d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.714848 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd6543699e254400b3516aece4486fe0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:33.715176 4384 catalog_manager.cc:5001] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): alter failed for tablet 3bd8dae5699c4725915814044d76a257 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]),no further retry: Not found: Tablet not found: 3bd8dae5699c4725915814044d76a257
W20250626 19:57:33.718605 4398 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 57) took 3129 ms (client timeout 1000 ms). Trace:
I20250626 19:57:33.722801 5433 ts_tablet_manager.cc:1918] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250626 19:57:33.719465 4398 rpcz_store.cc:269] 0626 19:57:30.589473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:30.628435 (+ 38962us) service_pool.cc:225] Handling call
0626 19:57:33.718515 (+3090080us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:31.494983 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:31.495699 (+ 716us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:31.495726 (+ 27us) write_op.cc:435] Acquired schema lock
0626 19:57:31.495743 (+ 17us) tablet.cc:660] Decoding operations
0626 19:57:31.496614 (+ 871us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:31.496664 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:31.496691 (+ 27us) tablet.cc:683] Acquiring locks for 9 operations
0626 19:57:31.497343 (+ 652us) tablet.cc:699] Row locks acquired
0626 19:57:31.497360 (+ 17us) write_op.cc:260] PREPARE: finished
0626 19:57:31.497498 (+ 138us) write_op.cc:270] Start()
0626 19:57:31.497589 (+ 91us) write_op.cc:276] Timestamp: P: 1750967851497515 usec, L: 0
0626 19:57:31.497608 (+ 19us) op_driver.cc:348] REPLICATION: starting
0626 19:57:31.505126 (+ 7518us) log.cc:844] Serialized 3546 byte log entry
0626 19:57:31.516100 (+ 10974us) op_driver.cc:464] REPLICATION: finished
0626 19:57:31.516467 (+ 367us) write_op.cc:301] APPLY: starting
0626 19:57:31.516524 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:31.517213 (+ 689us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:31.517307 (+ 94us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:31.517945 (+ 638us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:31.518116 (+ 171us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0626 19:57:31.518156 (+ 40us) write_op.cc:312] APPLY: finished
0626 19:57:31.518430 (+ 274us) log.cc:844] Serialized 70 byte log entry
0626 19:57:31.518571 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:31.518679 (+ 108us) write_op.cc:454] Released schema lock
0626 19:57:31.518735 (+ 56us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:31.650969 (+132234us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:31.651009 (+ 40us) write_op.cc:454] Released schema lock
Related trace 'op':
0626 19:57:31.656631 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:31.657085 (+ 454us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:31.657109 (+ 24us) write_op.cc:435] Acquired schema lock
0626 19:57:31.657124 (+ 15us) tablet.cc:660] Decoding operations
0626 19:57:31.657407 (+ 283us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:31.657447 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:31.657471 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:31.657620 (+ 149us) tablet.cc:699] Row locks acquired
0626 19:57:31.657632 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:31.657743 (+ 111us) write_op.cc:270] Start()
0626 19:57:31.657825 (+ 82us) write_op.cc:276] Timestamp: P: 1750967851657824 usec, L: 0
0626 19:57:31.657844 (+ 19us) op_driver.cc:348] REPLICATION: starting
0626 19:57:31.658368 (+ 524us) log.cc:844] Serialized 355 byte log entry
0626 19:57:31.730399 (+ 72031us) op_driver.cc:464] REPLICATION: finished
0626 19:57:31.732615 (+ 2216us) write_op.cc:301] APPLY: starting
0626 19:57:31.732666 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:31.732745 (+ 79us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:31.732820 (+ 75us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:31.732957 (+ 137us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:31.733075 (+ 118us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:31.733115 (+ 40us) write_op.cc:312] APPLY: finished
0626 19:57:31.733267 (+ 152us) log.cc:844] Serialized 22 byte log entry
0626 19:57:31.733387 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:31.733442 (+ 55us) write_op.cc:454] Released schema lock
0626 19:57:31.733471 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:31.733683 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:31.733710 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":210,"apply.run_cpu_time_us":2495,"apply.run_wall_time_us":3234,"num_ops":9,"prepare.queue_time_us":29424,"prepare.run_cpu_time_us":3641,"prepare.run_wall_time_us":155935,"raft.queue_time_us":78,"raft.run_cpu_time_us":639,"raft.run_wall_time_us":639,"replication_time_us":18401}],["op",{"apply.queue_time_us":2099,"apply.run_cpu_time_us":1027,"apply.run_wall_time_us":1026,"num_ops":1,"prepare.queue_time_us":86093,"prepare.run_cpu_time_us":2164,"prepare.run_wall_time_us":42040,"raft.queue_time_us":187,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":14,"replication_time_us":72479}]]}
I20250626 19:57:33.724108 4821 raft_consensus.cc:2466] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:33.726210 5417 raft_consensus.cc:491] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:33.726598 5417 raft_consensus.cc:513] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.727962 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 7391b9eaea3e426ba4118fc8b53b72a6 (DEFAULT_TABLE table=default.Table_e5f9396b41eb4c5eb98ec42ec48d7cdc [id=bad1d85ef3a347ab9abe78289306ca4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.728391 5417 leader_election.cc:290] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:33.729009 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7391b9eaea3e426ba4118fc8b53b72a6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:33.733004 4395 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 58) took 2092 ms (client timeout 1000 ms). Trace:
I20250626 19:57:33.734700 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 04b0f44c1b564fdf98408881678a90a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c03691d021e4ee19d5bb287ab633a3c) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:33.736608 4951 tablet_service.cc:1468] Processing CreateTablet for tablet 65af5ae255d74143976392bccc50f3ba (DEFAULT_TABLE table=default.Table_d2c793438d234509974deb53c4e61c8b [id=84b3a43b3e5b42849f8b343a97906309]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.736917 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c990e4eca5cf4a8e8295e9599cf5be29" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:33.737491 4819 raft_consensus.cc:2466] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:33.737798 4951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65af5ae255d74143976392bccc50f3ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:33.739300 4395 rpcz_store.cc:269] 0626 19:57:31.641001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:31.641171 (+ 170us) service_pool.cc:225] Handling call
0626 19:57:33.732925 (+2091754us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:32.878030 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:32.878601 (+ 571us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:32.878650 (+ 49us) write_op.cc:435] Acquired schema lock
0626 19:57:32.878672 (+ 22us) tablet.cc:660] Decoding operations
0626 19:57:32.879050 (+ 378us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:32.879107 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:32.879135 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:32.879290 (+ 155us) tablet.cc:699] Row locks acquired
0626 19:57:32.879313 (+ 23us) write_op.cc:260] PREPARE: finished
0626 19:57:32.879437 (+ 124us) write_op.cc:270] Start()
0626 19:57:32.879527 (+ 90us) write_op.cc:276] Timestamp: P: 1750967852879591 usec, L: 0
0626 19:57:32.879548 (+ 21us) op_driver.cc:348] REPLICATION: starting
0626 19:57:32.880175 (+ 627us) log.cc:844] Serialized 529 byte log entry
0626 19:57:32.890323 (+ 10148us) op_driver.cc:464] REPLICATION: finished
0626 19:57:32.891313 (+ 990us) write_op.cc:301] APPLY: starting
0626 19:57:32.891370 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:32.891459 (+ 89us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:32.891551 (+ 92us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:32.891731 (+ 180us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:32.891873 (+ 142us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:32.891913 (+ 40us) write_op.cc:312] APPLY: finished
0626 19:57:32.892079 (+ 166us) log.cc:844] Serialized 22 byte log entry
0626 19:57:32.892217 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:32.892278 (+ 61us) write_op.cc:454] Released schema lock
0626 19:57:32.892315 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:32.892546 (+ 231us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:32.892570 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":68609,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":238,"apply.run_cpu_time_us":1208,"apply.run_wall_time_us":1208,"num_ops":1,"prepare.queue_time_us":6019,"prepare.run_cpu_time_us":2503,"prepare.run_wall_time_us":2503,"raft.queue_time_us":3558,"raft.run_cpu_time_us":648,"raft.run_wall_time_us":648,"replication_time_us":10671}]]}
W20250626 19:57:33.738883 4399 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 68) took 1133 ms (client timeout 1000 ms). Trace:
W20250626 19:57:33.736958 4390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 67) took 2174 ms (client timeout 1000 ms). Trace:
W20250626 19:57:33.746863 4390 rpcz_store.cc:269] 0626 19:57:31.562196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:31.562358 (+ 162us) service_pool.cc:225] Handling call
0626 19:57:31.628484 (+ 66126us) spinlock_profiling.cc:243] Waited 61.6 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe7915557ee 00007fe78f26c624 00007fe78f26c5b2 00007fe78f26c4e1 00007fe78f26c45c 00007fe78f26b712 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:33.736878 (+2108394us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":123176,"reported_tablets":24,"spinlock_wait_cycles":141628800}
I20250626 19:57:33.748989 4390 master_service.cc:948] ReplaceTablet: received request to replace tablet af58f34e2248432fae5c084bd3388028 from {username='slave'} at 127.0.0.1:34796
W20250626 19:57:33.752233 4756 leader_election.cc:343] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:33.753461 4756 leader_election.cc:343] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:33.760267 4950 tablet_service.cc:1468] Processing CreateTablet for tablet 3c03691d021e4ee19d5bb287ab633a3c (DEFAULT_TABLE table=default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.761710 4950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c03691d021e4ee19d5bb287ab633a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.763474 5438 ts_tablet_manager.cc:1905] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:33.743090 4399 rpcz_store.cc:269] 0626 19:57:32.605760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:32.800810 (+195050us) service_pool.cc:225] Handling call
0626 19:57:33.738809 (+937999us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:33.191876 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:33.192368 (+ 492us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:33.192402 (+ 34us) write_op.cc:435] Acquired schema lock
0626 19:57:33.192425 (+ 23us) tablet.cc:660] Decoding operations
0626 19:57:33.192780 (+ 355us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:33.192840 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:33.192877 (+ 37us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:33.193093 (+ 216us) tablet.cc:699] Row locks acquired
0626 19:57:33.193116 (+ 23us) write_op.cc:260] PREPARE: finished
0626 19:57:33.193272 (+ 156us) write_op.cc:270] Start()
0626 19:57:33.193357 (+ 85us) write_op.cc:276] Timestamp: P: 1750967853193530 usec, L: 0
0626 19:57:33.193375 (+ 18us) op_driver.cc:348] REPLICATION: starting
0626 19:57:33.193853 (+ 478us) log.cc:844] Serialized 529 byte log entry
0626 19:57:33.206246 (+ 12393us) op_driver.cc:464] REPLICATION: finished
0626 19:57:33.206824 (+ 578us) write_op.cc:301] APPLY: starting
0626 19:57:33.206877 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:33.206978 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:33.207060 (+ 82us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:33.207226 (+ 166us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:33.207352 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:33.207394 (+ 42us) write_op.cc:312] APPLY: finished
0626 19:57:33.207547 (+ 153us) log.cc:844] Serialized 22 byte log entry
0626 19:57:33.207717 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.207778 (+ 61us) write_op.cc:454] Released schema lock
0626 19:57:33.207807 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:33.208017 (+ 210us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.208043 (+ 26us) write_op.cc:454] Released schema lock
Related trace 'op':
0626 19:57:33.582185 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:33.582617 (+ 432us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:33.582643 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:33.582654 (+ 11us) tablet.cc:660] Decoding operations
0626 19:57:33.582971 (+ 317us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:33.586228 (+ 3257us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:33.586267 (+ 39us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:33.586446 (+ 179us) tablet.cc:699] Row locks acquired
0626 19:57:33.586464 (+ 18us) write_op.cc:260] PREPARE: finished
0626 19:57:33.586594 (+ 130us) write_op.cc:270] Start()
0626 19:57:33.586695 (+ 101us) write_op.cc:276] Timestamp: P: 1750967853586671 usec, L: 0
0626 19:57:33.586711 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:57:33.587333 (+ 622us) log.cc:844] Serialized 355 byte log entry
0626 19:57:33.599243 (+ 11910us) op_driver.cc:464] REPLICATION: finished
0626 19:57:33.599590 (+ 347us) write_op.cc:301] APPLY: starting
0626 19:57:33.599643 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:33.599734 (+ 91us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:33.599805 (+ 71us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:33.599997 (+ 192us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:33.600123 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:33.600161 (+ 38us) write_op.cc:312] APPLY: finished
0626 19:57:33.600321 (+ 160us) log.cc:844] Serialized 22 byte log entry
0626 19:57:33.600451 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.600507 (+ 56us) write_op.cc:454] Released schema lock
0626 19:57:33.600535 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:33.600756 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.600777 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":99136,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":386,"apply.run_cpu_time_us":1161,"apply.run_wall_time_us":1160,"num_ops":1,"prepare.queue_time_us":249,"prepare.run_cpu_time_us":2362,"prepare.run_wall_time_us":2361,"raft.queue_time_us":4059,"raft.run_cpu_time_us":687,"raft.run_wall_time_us":689,"replication_time_us":12760}],["op",{"apply.queue_time_us":218,"apply.run_cpu_time_us":1108,"apply.run_wall_time_us":1107,"mutex_wait_us":19938,"num_ops":1,"prepare.queue_time_us":20125,"prepare.run_cpu_time_us":2244,"prepare.run_wall_time_us":8220,"raft.queue_time_us":75,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":12,"replication_time_us":12438}]]}
I20250626 19:57:33.765959 4399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_89c3869cb74248c4af47e8eaa3371b16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:33.768422 5429 raft_consensus.cc:491] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:33.769091 5429 raft_consensus.cc:513] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:33.771265 4398 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_dafe34566c694a47a3d9321791db3fb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:33.772861 4398 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dafe34566c694a47a3d9321791db3fb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:33.774645 4949 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e453a4f8ca44f4a940928d9f4fcd5be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:31 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:33.648109 4397 rpcz_store.cc:276] 0626 19:57:32.328193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.484864 (+1156671us) service_pool.cc:225] Handling call
0626 19:57:33.487890 (+ 3026us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:33.489364 (+ 1474us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:33.646832 (+157468us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.646857 (+ 25us) write_op.cc:454] Released schema lock
0626 19:57:33.647056 (+ 199us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:33.647340 (+ 284us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:33.647612 (+ 272us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:33.576110 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:33.576622 (+ 512us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:33.576652 (+ 30us) write_op.cc:435] Acquired schema lock
0626 19:57:33.576671 (+ 19us) tablet.cc:660] Decoding operations
0626 19:57:33.577050 (+ 379us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:33.577107 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:33.577146 (+ 39us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:33.577393 (+ 247us) tablet.cc:699] Row locks acquired
0626 19:57:33.577413 (+ 20us) write_op.cc:260] PREPARE: finished
0626 19:57:33.577568 (+ 155us) write_op.cc:270] Start()
0626 19:57:33.577699 (+ 131us) write_op.cc:276] Timestamp: P: 1750967853577617 usec, L: 0
0626 19:57:33.577739 (+ 40us) op_driver.cc:348] REPLICATION: starting
0626 19:57:33.578343 (+ 604us) log.cc:844] Serialized 432 byte log entry
0626 19:57:33.592180 (+ 13837us) op_driver.cc:464] REPLICATION: finished
0626 19:57:33.593079 (+ 899us) write_op.cc:301] APPLY: starting
0626 19:57:33.593135 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:33.593232 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:33.593309 (+ 77us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:33.593664 (+ 355us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:33.593835 (+ 171us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:33.593877 (+ 42us) write_op.cc:312] APPLY: finished
0626 19:57:33.594041 (+ 164us) log.cc:844] Serialized 28 byte log entry
0626 19:57:33.594197 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.594259 (+ 62us) write_op.cc:454] Released schema lock
0626 19:57:33.594293 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":729,"apply.run_cpu_time_us":1379,"apply.run_wall_time_us":1377,"num_ops":2,"prepare.queue_time_us":85616,"prepare.run_cpu_time_us":2636,"prepare.run_wall_time_us":2636,"raft.queue_time_us":3413,"raft.run_cpu_time_us":661,"raft.run_wall_time_us":660,"replication_time_us":14336}]]}
I20250626 19:57:33.727595 4650 tablet_service.cc:1515] Processing DeleteTablet for tablet afd1fafb53434957b002fad5c77431d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:33.728060 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet df3da12971654d9abc2ee8840c4f1592 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:30 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:33.729790 4649 tablet_service.cc:1515] Processing DeleteTablet for tablet 5785508e37624f02bc2f6cc93007e3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:28 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:33.778213 5433 log.cc:1199] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/1e453a4f8ca44f4a940928d9f4fcd5be
I20250626 19:57:33.730232 4648 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a8f494f8614e7ebb8ca880b6c80e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:33.730276 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c990e4eca5cf4a8e8295e9599cf5be29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:33.779017 4675 raft_consensus.cc:2466] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:33.779175 5433 ts_tablet_manager.cc:1939] T 1e453a4f8ca44f4a940928d9f4fcd5be P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:33.780282 4755 leader_election.cc:304] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:33.730788 4654 tablet_service.cc:1515] Processing DeleteTablet for tablet ea7374ad6ba64e46b7df9852fa33f145 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:33.780362 5455 consensus_queue.cc:1035] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:33.743173 5444 raft_consensus.cc:491] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:33.781287 5444 raft_consensus.cc:513] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.782727 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 1e453a4f8ca44f4a940928d9f4fcd5be (table default.Table_f579d2c76b104fc88466bcdcd54d2603 [id=049afd5ef383451fa96431d71b4f2ae4]) successfully deleted
I20250626 19:57:33.783167 5444 leader_election.cc:290] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:33.784718 5429 leader_election.cc:290] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
W20250626 19:57:33.785365 4392 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e20ea7c17c34fbd9c117275e3c14466 consensus state has no leader
W20250626 19:57:33.785677 4755 leader_election.cc:343] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Not found: Tablet not found: 61fcd55f6d7b4a369383d4f73d5f483d
I20250626 19:57:33.785804 5410 tablet_bootstrap.cc:654] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:33.785904 4392 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34796 (request call id 68) took 1202 ms. Trace:
I20250626 19:57:33.786185 5416 raft_consensus.cc:2802] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:33.786506 5416 raft_consensus.cc:491] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:33.786839 5416 raft_consensus.cc:3058] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:33.744450 4609 leader_election.cc:304] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:33.789386 5444 raft_consensus.cc:2802] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:33.789736 5444 raft_consensus.cc:491] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:33.790043 5444 raft_consensus.cc:3058] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:33.792086 5366 tablet_bootstrap.cc:654] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:33.796418 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 52062.1us
I20250626 19:57:33.797508 5433 tablet_replica.cc:331] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:33.800240 5433 raft_consensus.cc:2241] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:33.801265 5433 raft_consensus.cc:2270] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:33.806872 5410 tablet_bootstrap.cc:492] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:33.807433 5410 ts_tablet_manager.cc:1397] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.264s user 0.010s sys 0.005s
I20250626 19:57:33.786190 4392 rpcz_store.cc:276] 0626 19:57:32.583909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.498501 (+914592us) service_pool.cc:225] Handling call
0626 19:57:33.499620 (+ 1119us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:33.502492 (+ 2872us) catalog_manager.cc:3683] Apply alter schema
0626 19:57:33.503344 (+ 852us) catalog_manager.cc:3715] Acquired catalog manager lock
0626 19:57:33.503660 (+ 316us) catalog_manager.cc:3873] Updating metadata on disk
0626 19:57:33.784836 (+281176us) catalog_manager.cc:3904] Committing alterations to in-memory state
0626 19:57:33.785840 (+ 1004us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:33.679258 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:33.679741 (+ 483us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:33.679769 (+ 28us) write_op.cc:435] Acquired schema lock
0626 19:57:33.679781 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:33.680056 (+ 275us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:33.680102 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:33.680128 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:33.680283 (+ 155us) tablet.cc:699] Row locks acquired
0626 19:57:33.680295 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:33.680431 (+ 136us) write_op.cc:270] Start()
0626 19:57:33.680532 (+ 101us) write_op.cc:276] Timestamp: P: 1750967853680535 usec, L: 0
0626 19:57:33.680548 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:57:33.682312 (+ 1764us) log.cc:844] Serialized 405 byte log entry
0626 19:57:33.781678 (+ 99366us) op_driver.cc:464] REPLICATION: finished
0626 19:57:33.782094 (+ 416us) write_op.cc:301] APPLY: starting
0626 19:57:33.782148 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:33.782239 (+ 91us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:33.782337 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:33.783288 (+ 951us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:33.783432 (+ 144us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:33.783475 (+ 43us) write_op.cc:312] APPLY: finished
0626 19:57:33.783647 (+ 172us) log.cc:844] Serialized 23 byte log entry
0626 19:57:33.783788 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.783846 (+ 58us) write_op.cc:454] Released schema lock
0626 19:57:33.783879 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:33.784098 (+ 219us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.784120 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":263,"apply.run_cpu_time_us":1903,"apply.run_wall_time_us":1967,"mutex_wait_us":24721,"num_ops":1,"prepare.queue_time_us":24891,"prepare.run_cpu_time_us":2338,"prepare.run_wall_time_us":38046,"raft.queue_time_us":195,"raft.run_cpu_time_us":664,"raft.run_wall_time_us":786,"replication_time_us":101027}]]}
I20250626 19:57:33.810338 5410 raft_consensus.cc:357] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.803721 5433 ts_tablet_manager.cc:1905] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:33.811232 5410 raft_consensus.cc:383] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:33.816416 4390 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet af58f34e2248432fae5c084bd3388028
I20250626 19:57:33.816809 5410 raft_consensus.cc:738] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:33.818002 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet af58f34e2248432fae5c084bd3388028 with delete_type TABLET_DATA_DELETED (replaced by tablet e60b4d53c40c4dffab928f0e3fb4a80b) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:33.819345 4390 catalog_manager.cc:6652] ReplaceTablet: tablet af58f34e2248432fae5c084bd3388028 deleted and replaced by tablet e60b4d53c40c4dffab928f0e3fb4a80b
I20250626 19:57:33.819554 4957 tablet_service.cc:1468] Processing CreateTablet for tablet 7facb77eb579448f9533d285cba8650e (DEFAULT_TABLE table=default.Table_e72dff976c234b9fa5819cb8c6e31d73 [id=e4b13e1cb6c143339891b9deda4ee7bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:33.820129 4799 tablet_service.cc:1515] Processing DeleteTablet for tablet af58f34e2248432fae5c084bd3388028 with delete_type TABLET_DATA_DELETED (replaced by tablet e60b4d53c40c4dffab928f0e3fb4a80b) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:33.820700 4390 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34796 (request call id 70) took 1191 ms. Trace:
I20250626 19:57:33.819202 5410 consensus_queue.cc:260] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.814229 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f91a78f6d474133976242ea8d259e29" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:33.811296 5444 raft_consensus.cc:513] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.827888 5410 ts_tablet_manager.cc:1428] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.020s user 0.003s sys 0.003s
I20250626 19:57:33.828838 5410 tablet_bootstrap.cc:492] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:33.829242 5444 leader_election.cc:290] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:33.820282 4397 master_service.cc:948] ReplaceTablet: received request to replace tablet 499f6ae3262940b4859c14a53722482f from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:33.840561 5410 tablet_bootstrap.cc:654] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:33.849665 5433 ts_tablet_manager.cc:1918] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:33.850373 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fda659c30c44c8ba402ac5c7f3598e0" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:33.851012 4820 raft_consensus.cc:2466] T 2fda659c30c44c8ba402ac5c7f3598e0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:33.852962 5433 log.cc:1199] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/4d8ffa9b658c413fa06d02ee6635dd1b
I20250626 19:57:33.853813 5433 ts_tablet_manager.cc:1939] T 4d8ffa9b658c413fa06d02ee6635dd1b P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:33.856784 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 4d8ffa9b658c413fa06d02ee6635dd1b (table default.Table_681222687426414bb0838e947d315979 [id=4419bc765f3a4ff1a104bb656bf642ea]) successfully deleted
W20250626 19:57:33.767580 4399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89c3869cb74248c4af47e8eaa3371b16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:33.821240 4390 rpcz_store.cc:276] 0626 19:57:32.628607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.748814 (+1120207us) service_pool.cc:225] Handling call
0626 19:57:33.819842 (+ 71028us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:33.753140 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:33.753591 (+ 451us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:33.753622 (+ 31us) write_op.cc:435] Acquired schema lock
0626 19:57:33.753635 (+ 13us) tablet.cc:660] Decoding operations
0626 19:57:33.753962 (+ 327us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:33.753999 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:33.754020 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:33.754211 (+ 191us) tablet.cc:699] Row locks acquired
0626 19:57:33.754226 (+ 15us) write_op.cc:260] PREPARE: finished
0626 19:57:33.754331 (+ 105us) write_op.cc:270] Start()
0626 19:57:33.754445 (+ 114us) write_op.cc:276] Timestamp: P: 1750967853754485 usec, L: 0
0626 19:57:33.754462 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:33.754930 (+ 468us) log.cc:844] Serialized 659 byte log entry
0626 19:57:33.800322 (+ 45392us) op_driver.cc:464] REPLICATION: finished
0626 19:57:33.800706 (+ 384us) write_op.cc:301] APPLY: starting
0626 19:57:33.800755 (+ 49us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:33.800855 (+ 100us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:33.800938 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:33.801217 (+ 279us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:33.801414 (+ 197us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:33.801454 (+ 40us) write_op.cc:312] APPLY: finished
0626 19:57:33.801605 (+ 151us) log.cc:844] Serialized 29 byte log entry
0626 19:57:33.801738 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.801799 (+ 61us) write_op.cc:454] Released schema lock
0626 19:57:33.801828 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:33.802020 (+ 192us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.802043 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":248,"apply.run_cpu_time_us":1281,"apply.run_wall_time_us":1279,"num_ops":2,"prepare.queue_time_us":95,"prepare.run_cpu_time_us":2084,"prepare.run_wall_time_us":3392,"raft.queue_time_us":188,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":13,"replication_time_us":45773}]]}
I20250626 19:57:33.861943 5433 tablet_replica.cc:331] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:33.862591 4609 leader_election.cc:304] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:33.863678 5448 raft_consensus.cc:2802] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:33.863746 5433 raft_consensus.cc:2241] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:33.864146 5448 raft_consensus.cc:695] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:33.865535 4609 leader_election.cc:304] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:33.866195 5368 raft_consensus.cc:2802] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:33.865763 5448 consensus_queue.cc:237] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.866523 5368 raft_consensus.cc:491] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:33.866895 5368 raft_consensus.cc:3058] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:33.870698 5433 raft_consensus.cc:2270] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 19:57:33.871315 4391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 59) took 1173 ms (client timeout 1000 ms). Trace:
I20250626 19:57:33.873965 5433 ts_tablet_manager.cc:1905] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:33.676200 4393 rpcz_store.cc:276] 0626 19:57:32.607395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.512440 (+905045us) service_pool.cc:225] Handling call
0626 19:57:33.516312 (+ 3872us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:33.518013 (+ 1701us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:33.673360 (+155347us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.673388 (+ 28us) write_op.cc:454] Released schema lock
0626 19:57:33.674221 (+ 833us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:33.675382 (+ 1161us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:33.675692 (+ 310us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:33.654514 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:33.655032 (+ 518us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:33.655061 (+ 29us) write_op.cc:435] Acquired schema lock
0626 19:57:33.655075 (+ 14us) tablet.cc:660] Decoding operations
0626 19:57:33.655395 (+ 320us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:33.655441 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:33.655478 (+ 37us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:33.655720 (+ 242us) tablet.cc:699] Row locks acquired
0626 19:57:33.655738 (+ 18us) write_op.cc:260] PREPARE: finished
0626 19:57:33.655990 (+ 252us) write_op.cc:270] Start()
0626 19:57:33.656086 (+ 96us) write_op.cc:276] Timestamp: P: 1750967853656095 usec, L: 0
0626 19:57:33.656109 (+ 23us) op_driver.cc:348] REPLICATION: starting
0626 19:57:33.656697 (+ 588us) log.cc:844] Serialized 433 byte log entry
0626 19:57:33.666149 (+ 9452us) op_driver.cc:464] REPLICATION: finished
0626 19:57:33.666612 (+ 463us) write_op.cc:301] APPLY: starting
0626 19:57:33.666686 (+ 74us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:33.666843 (+ 157us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:33.666936 (+ 93us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:33.667311 (+ 375us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:33.667471 (+ 160us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:33.667518 (+ 47us) write_op.cc:312] APPLY: finished
0626 19:57:33.667698 (+ 180us) log.cc:844] Serialized 29 byte log entry
0626 19:57:33.667840 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.667913 (+ 73us) write_op.cc:454] Released schema lock
0626 19:57:33.667945 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":299,"apply.run_cpu_time_us":1515,"apply.run_wall_time_us":1513,"num_ops":2,"prepare.queue_time_us":418,"prepare.run_cpu_time_us":2514,"prepare.run_wall_time_us":5222,"raft.queue_time_us":208,"raft.run_cpu_time_us":652,"raft.run_wall_time_us":656,"replication_time_us":9942}]]}
I20250626 19:57:33.877224 4393 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_fc5a0fda8b274c989cb3b9189b6a838d" } modify_external_catalogs: true
I20250626 19:57:33.873878 5368 raft_consensus.cc:513] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.876710 5410 tablet_bootstrap.cc:492] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:33.821591 4949 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d8ffa9b658c413fa06d02ee6635dd1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:31 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:33.881004 5410 ts_tablet_manager.cc:1397] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.052s user 0.013s sys 0.003s
I20250626 19:57:33.827807 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30f1d01851be485f9d766784855ba627" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:33.831292 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e20ea7c17c34fbd9c117275e3c14466" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:33.833514 4974 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c990e4eca5cf4a8e8295e9599cf5be29" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:33.834694 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fda659c30c44c8ba402ac5c7f3598e0" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:33.884352 5368 leader_election.cc:290] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:33.884366 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet 512dc911bfce49b29c6fc8cd90523723 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:33.885151 5410 raft_consensus.cc:357] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:33.885914 5410 raft_consensus.cc:383] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:33.886253 5410 raft_consensus.cc:738] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:33.842391 5353 tablet_bootstrap.cc:492] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:33.848764 4948 tablet_service.cc:1515] Processing DeleteTablet for tablet fa93fa6c652f4aca848e1e4910fb8175 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fda659c30c44c8ba402ac5c7f3598e0) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:33.849815 4947 tablet_service.cc:1515] Processing DeleteTablet for tablet df3da12971654d9abc2ee8840c4f1592 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:30 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:33.851334 4946 tablet_service.cc:1515] Processing DeleteTablet for tablet ea7374ad6ba64e46b7df9852fa33f145 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:33.852365 4945 tablet_service.cc:1515] Processing DeleteTablet for tablet 5785508e37624f02bc2f6cc93007e3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:28 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:33.854056 4944 tablet_service.cc:1515] Processing DeleteTablet for tablet afd1fafb53434957b002fad5c77431d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:33.859802 4943 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bd8dae5699c4725915814044d76a257 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c0338c05a974e0f96865398722c5e8f) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:33.889561 5353 ts_tablet_manager.cc:1397] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.767s user 0.018s sys 0.000s
I20250626 19:57:33.888962 5410 consensus_queue.cc:260] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:33.865610 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:33.896165 5353 raft_consensus.cc:357] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.896997 5353 raft_consensus.cc:383] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:33.897344 5353 raft_consensus.cc:738] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:33.886309 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bd8dae5699c4725915814044d76a257 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c0338c05a974e0f96865398722c5e8f) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:33.900065 5410 ts_tablet_manager.cc:1428] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.019s user 0.007s sys 0.001s
I20250626 19:57:33.901010 5410 tablet_bootstrap.cc:492] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:33.900942 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e453a4f8ca44f4a940928d9f4fcd5be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:31 UTC) from {username='slave'} at 127.0.0.1:49550
W20250626 19:57:33.901430 4394 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 74) took 1324 ms (client timeout 1000 ms). Trace:
I20250626 19:57:33.901738 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d8ffa9b658c413fa06d02ee6635dd1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:31 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:33.910719 5416 raft_consensus.cc:513] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.915663 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c990e4eca5cf4a8e8295e9599cf5be29" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:33.915995 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet 04b0f44c1b564fdf98408881678a90a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c03691d021e4ee19d5bb287ab633a3c) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:33.917825 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7facb77eb579448f9533d285cba8650e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:33.920626 5410 tablet_bootstrap.cc:654] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:33.928280 4396 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 73) took 2374 ms (client timeout 1000 ms). Trace:
W20250626 19:57:33.928718 4396 rpcz_store.cc:269] 0626 19:57:31.554103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:31.554489 (+ 386us) service_pool.cc:225] Handling call
0626 19:57:31.631868 (+ 77379us) spinlock_profiling.cc:243] Waited 72.9 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe7915557ee 00007fe78f26c624 00007fe78f26c5b2 00007fe78f26c4e1 00007fe78f26c45c 00007fe78f26b712 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:33.928208 (+2296340us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":25,"spinlock_wait_cycles":167676288}
I20250626 19:57:33.930703 4396 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_ad656f771f554bc98b2dc4db559e3acb" } new_table_name: "default.Table_c022e4f4da8b477a8606e9c34729c8eb" modify_external_catalogs: true
I20250626 19:57:33.935204 5410 tablet_bootstrap.cc:492] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:33.935580 5410 ts_tablet_manager.cc:1397] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.002s
I20250626 19:57:33.940232 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fda659c30c44c8ba402ac5c7f3598e0" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:33.940685 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c990e4eca5cf4a8e8295e9599cf5be29" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:33.941417 4819 raft_consensus.cc:2391] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 93f885c6802341c889a40e9c20fbfff1 in this term.
I20250626 19:57:33.941260 5410 raft_consensus.cc:357] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:33.941855 5410 raft_consensus.cc:383] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:33.942099 5410 raft_consensus.cc:738] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:33.940801 4820 raft_consensus.cc:3058] T 2fda659c30c44c8ba402ac5c7f3598e0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:33.945531 5410 consensus_queue.cc:260] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:33.948985 5410 ts_tablet_manager.cc:1428] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250626 19:57:33.949805 5410 tablet_bootstrap.cc:492] T 33a75d223c274c049cf25f79fcc4b961 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:33.951188 4820 raft_consensus.cc:2466] T 2fda659c30c44c8ba402ac5c7f3598e0 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:33.898051 5353 consensus_queue.cc:260] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.953312 4942 tablet_service.cc:1515] Processing DeleteTablet for tablet 612f87796cfa44ddbb19b02797031db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:33.954212 5353 ts_tablet_manager.cc:1428] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.061s user 0.006s sys 0.000s
I20250626 19:57:33.954502 4942 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a8f494f8614e7ebb8ca880b6c80e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:33.955964 4942 tablet_service.cc:1515] Processing DeleteTablet for tablet 512dc911bfce49b29c6fc8cd90523723 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:33.956979 4942 tablet_service.cc:1515] Processing DeleteTablet for tablet af58f34e2248432fae5c084bd3388028 with delete_type TABLET_DATA_DELETED (replaced by tablet e60b4d53c40c4dffab928f0e3fb4a80b) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:33.969488 4609 leader_election.cc:304] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:33.958132 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30f1d01851be485f9d766784855ba627" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:33.976382 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a31f1a7739544706bff2a500d6026086" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:33.977627 4384 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 5e20ea7c17c34fbd9c117275e3c14466 (table default.Table_84725c81a9a9432c8232f87fb7627f6a [id=089059dab32a41179048a6eb4473a99a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5e20ea7c17c34fbd9c117275e3c14466 consensus state has no leader
I20250626 19:57:33.978088 4975 raft_consensus.cc:3058] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Advancing to term 3
I20250626 19:57:33.985378 5366 tablet_bootstrap.cc:492] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:33.986589 5366 ts_tablet_manager.cc:1397] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.298s user 0.009s sys 0.001s
I20250626 19:57:33.988332 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c990e4eca5cf4a8e8295e9599cf5be29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:33.993067 4676 raft_consensus.cc:2391] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 93f885c6802341c889a40e9c20fbfff1 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:33.990957 5410 tablet_bootstrap.cc:654] T 33a75d223c274c049cf25f79fcc4b961 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:33.997020 5448 raft_consensus.cc:2802] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:33.997570 5448 raft_consensus.cc:695] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:33.998728 5448 consensus_queue.cc:237] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
W20250626 19:57:33.901822 4394 rpcz_store.cc:269] 0626 19:57:32.576778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:32.793681 (+216903us) service_pool.cc:225] Handling call
0626 19:57:33.901361 (+1107680us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:33.332377 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:33.332849 (+ 472us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:33.332878 (+ 29us) write_op.cc:435] Acquired schema lock
0626 19:57:33.332891 (+ 13us) tablet.cc:660] Decoding operations
0626 19:57:33.333349 (+ 458us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:33.333412 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:33.333457 (+ 45us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:33.333764 (+ 307us) tablet.cc:699] Row locks acquired
0626 19:57:33.333776 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:33.333915 (+ 139us) write_op.cc:270] Start()
0626 19:57:33.334011 (+ 96us) write_op.cc:276] Timestamp: P: 1750967853333896 usec, L: 0
0626 19:57:33.334028 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:33.334575 (+ 547us) log.cc:844] Serialized 906 byte log entry
0626 19:57:33.344526 (+ 9951us) op_driver.cc:464] REPLICATION: finished
0626 19:57:33.344917 (+ 391us) write_op.cc:301] APPLY: starting
0626 19:57:33.344971 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:33.345076 (+ 105us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:33.345154 (+ 78us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:33.345353 (+ 199us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:33.345483 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:33.345524 (+ 41us) write_op.cc:312] APPLY: finished
0626 19:57:33.345685 (+ 161us) log.cc:844] Serialized 28 byte log entry
0626 19:57:33.345823 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.345900 (+ 77us) write_op.cc:454] Released schema lock
0626 19:57:33.345930 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:33.346147 (+ 217us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.346170 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":54623,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":244,"apply.run_cpu_time_us":1200,"apply.run_wall_time_us":1199,"num_ops":2,"prepare.queue_time_us":113,"prepare.run_cpu_time_us":2649,"prepare.run_wall_time_us":2648,"raft.queue_time_us":3802,"raft.run_cpu_time_us":693,"raft.run_wall_time_us":693,"replication_time_us":10400}]]}
I20250626 19:57:33.862522 4390 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_a2a4a2c9c59248dfb73f7db218b0d982" } modify_external_catalogs: true
I20250626 19:57:33.960048 4393 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_fc5a0fda8b274c989cb3b9189b6a838d" } modify_external_catalogs: true
W20250626 19:57:33.871785 4391 rpcz_store.cc:269] 0626 19:57:32.697927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:32.996093 (+298166us) service_pool.cc:225] Handling call
0626 19:57:33.637233 (+641140us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.637259 (+ 26us) write_op.cc:454] Released schema lock
0626 19:57:33.871237 (+233978us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:33.590554 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:33.601580 (+ 11026us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:33.601604 (+ 24us) write_op.cc:435] Acquired schema lock
0626 19:57:33.601615 (+ 11us) tablet.cc:660] Decoding operations
0626 19:57:33.601900 (+ 285us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:33.601940 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:33.601963 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:33.602104 (+ 141us) tablet.cc:699] Row locks acquired
0626 19:57:33.602116 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:33.602230 (+ 114us) write_op.cc:270] Start()
0626 19:57:33.602308 (+ 78us) write_op.cc:276] Timestamp: P: 1750967853602323 usec, L: 0
0626 19:57:33.602322 (+ 14us) op_driver.cc:348] REPLICATION: starting
0626 19:57:33.602838 (+ 516us) log.cc:844] Serialized 529 byte log entry
0626 19:57:33.624105 (+ 21267us) op_driver.cc:464] REPLICATION: finished
0626 19:57:33.624560 (+ 455us) write_op.cc:301] APPLY: starting
0626 19:57:33.624620 (+ 60us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:33.624735 (+ 115us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:33.624844 (+ 109us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:33.625005 (+ 161us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:33.625141 (+ 136us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:33.625188 (+ 47us) write_op.cc:312] APPLY: finished
0626 19:57:33.625371 (+ 183us) log.cc:844] Serialized 22 byte log entry
0626 19:57:33.625515 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.625589 (+ 74us) write_op.cc:454] Released schema lock
0626 19:57:33.625622 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":293,"apply.run_cpu_time_us":1248,"apply.run_wall_time_us":1247,"mutex_wait_us":6118,"num_ops":1,"prepare.queue_time_us":14549,"prepare.run_cpu_time_us":2191,"prepare.run_wall_time_us":22469,"raft.queue_time_us":89,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":14,"replication_time_us":21683}]]}
I20250626 19:57:34.015530 5429 consensus_queue.cc:1035] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:34.028331 5408 raft_consensus.cc:491] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:34.028782 5408 raft_consensus.cc:513] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:33.992009 5416 leader_election.cc:290] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:34.033944 5366 raft_consensus.cc:357] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.038028 5410 tablet_bootstrap.cc:492] T 33a75d223c274c049cf25f79fcc4b961 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:34.038465 5410 ts_tablet_manager.cc:1397] T 33a75d223c274c049cf25f79fcc4b961 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.089s user 0.005s sys 0.005s
I20250626 19:57:34.038148 5477 tablet.cc:1722] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:34.046571 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fda659c30c44c8ba402ac5c7f3598e0" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:34.047410 4821 raft_consensus.cc:2391] T 2fda659c30c44c8ba402ac5c7f3598e0 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:34.051229 4397 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 499f6ae3262940b4859c14a53722482f
I20250626 19:57:34.052945 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet 499f6ae3262940b4859c14a53722482f with delete_type TABLET_DATA_DELETED (replaced by tablet 6b8473259485460783f62f8193d5b02c) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:34.054603 4397 catalog_manager.cc:6652] ReplaceTablet: tablet 499f6ae3262940b4859c14a53722482f deleted and replaced by tablet 6b8473259485460783f62f8193d5b02c
I20250626 19:57:34.055594 4390 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_a2a4a2c9c59248dfb73f7db218b0d982" } modify_external_catalogs: true
I20250626 19:57:34.034658 5366 raft_consensus.cc:383] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:34.058867 5366 raft_consensus.cc:738] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:34.030632 5408 leader_election.cc:290] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:34.065001 4942 tablet_service.cc:1515] Processing DeleteTablet for tablet 499f6ae3262940b4859c14a53722482f with delete_type TABLET_DATA_DELETED (replaced by tablet 6b8473259485460783f62f8193d5b02c) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:34.069128 4394 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_605c8b4275614ae2a45a5ad6224f3a43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:34.075042 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 499f6ae3262940b4859c14a53722482f with delete_type TABLET_DATA_DELETED (replaced by tablet 6b8473259485460783f62f8193d5b02c) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:34.075052 5481 tablet.cc:1722] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:34.079391 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_c60978a633d54c628ce6fbb9e0cdac48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:34.081368 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c60978a633d54c628ce6fbb9e0cdac48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:34.081967 4397 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_84725c81a9a9432c8232f87fb7627f6a" } new_table_name: "default.Table_c0263194234444158f569a2abbfa91b5" modify_external_catalogs: true
I20250626 19:57:34.061094 5410 raft_consensus.cc:357] T 33a75d223c274c049cf25f79fcc4b961 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.086433 5433 ts_tablet_manager.cc:1918] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:34.087031 5433 log.cc:1199] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/afd1fafb53434957b002fad5c77431d4
I20250626 19:57:34.088117 5433 ts_tablet_manager.cc:1939] T afd1fafb53434957b002fad5c77431d4 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:34.080008 5366 consensus_queue.cc:260] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.094461 5366 ts_tablet_manager.cc:1428] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.107s user 0.006s sys 0.000s
I20250626 19:57:34.095045 4398 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 72) took 1086 ms. Trace:
I20250626 19:57:34.095418 4398 rpcz_store.cc:276] 0626 19:57:33.008981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.770905 (+761924us) service_pool.cc:225] Handling call
0626 19:57:33.774149 (+ 3244us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:33.918023 (+143874us) spinlock_profiling.cc:243] Waited 24.7 ms on lock 0x7b54000023e8. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78869e9a1 00007fe7887fdfee 00007fe79147a3a7 00007fe791437876 00007fe791434f57 00007fe79143856e 00007fe791556d6f 00007fe78f2736a4 00007fe78f273632 00007fe78f273561 00007fe78f2734dc
0626 19:57:33.918528 (+ 505us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:33.983440 (+ 64912us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:34.094663 (+111223us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:34.094988 (+ 325us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:33.920260 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:33.920628 (+ 368us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:33.920653 (+ 25us) write_op.cc:435] Acquired schema lock
0626 19:57:33.920669 (+ 16us) tablet.cc:660] Decoding operations
0626 19:57:33.920927 (+ 258us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:33.920966 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:33.920989 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:33.921160 (+ 171us) tablet.cc:699] Row locks acquired
0626 19:57:33.921173 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:57:33.921273 (+ 100us) write_op.cc:270] Start()
0626 19:57:33.921359 (+ 86us) write_op.cc:276] Timestamp: P: 1750967853921281 usec, L: 0
0626 19:57:33.921375 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:57:33.921797 (+ 422us) log.cc:844] Serialized 433 byte log entry
0626 19:57:33.932374 (+ 10577us) op_driver.cc:464] REPLICATION: finished
0626 19:57:33.932762 (+ 388us) write_op.cc:301] APPLY: starting
0626 19:57:33.932816 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:33.932919 (+ 103us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:33.933000 (+ 81us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:33.933369 (+ 369us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:33.933497 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:33.933540 (+ 43us) write_op.cc:312] APPLY: finished
0626 19:57:33.933722 (+ 182us) log.cc:844] Serialized 29 byte log entry
0626 19:57:33.933861 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.933930 (+ 69us) write_op.cc:454] Released schema lock
0626 19:57:33.933961 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:33.934171 (+ 210us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.934194 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":56758144,"child_traces":[["op",{"apply.queue_time_us":251,"apply.run_cpu_time_us":1373,"apply.run_wall_time_us":1372,"num_ops":2,"prepare.queue_time_us":131,"prepare.run_cpu_time_us":1827,"prepare.run_wall_time_us":1828,"raft.queue_time_us":2817,"raft.run_cpu_time_us":615,"raft.run_wall_time_us":615,"replication_time_us":10897}]]}
I20250626 19:57:34.102023 5366 tablet_bootstrap.cc:492] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:34.099327 5477 tablet.cc:1722] T 44960f8db08942879cde6f96e9c620da P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250626 19:57:34.100436 4399 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 71) took 1106 ms. Trace:
I20250626 19:57:34.103480 4399 rpcz_store.cc:276] 0626 19:57:32.993785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.765591 (+771806us) service_pool.cc:225] Handling call
0626 19:57:33.958820 (+193229us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:33.960357 (+ 1537us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:33.995037 (+ 34680us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.995066 (+ 29us) write_op.cc:454] Released schema lock
0626 19:57:33.995308 (+ 242us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:34.100113 (+104805us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:34.100380 (+ 267us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:33.961640 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:33.962006 (+ 366us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:33.962028 (+ 22us) write_op.cc:435] Acquired schema lock
0626 19:57:33.962041 (+ 13us) tablet.cc:660] Decoding operations
0626 19:57:33.962313 (+ 272us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:33.962360 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:33.962383 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:33.962578 (+ 195us) tablet.cc:699] Row locks acquired
0626 19:57:33.962591 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:57:33.962702 (+ 111us) write_op.cc:270] Start()
0626 19:57:33.962806 (+ 104us) write_op.cc:276] Timestamp: P: 1750967853962724 usec, L: 0
0626 19:57:33.962822 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:57:33.963330 (+ 508us) log.cc:844] Serialized 433 byte log entry
0626 19:57:33.975724 (+ 12394us) op_driver.cc:464] REPLICATION: finished
0626 19:57:33.976117 (+ 393us) write_op.cc:301] APPLY: starting
0626 19:57:33.976171 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:33.976271 (+ 100us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:33.976389 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:33.976813 (+ 424us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:33.976978 (+ 165us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:33.977023 (+ 45us) write_op.cc:312] APPLY: finished
0626 19:57:33.977218 (+ 195us) log.cc:844] Serialized 29 byte log entry
0626 19:57:33.977374 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:33.977437 (+ 63us) write_op.cc:454] Released schema lock
0626 19:57:33.977471 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":248,"apply.run_cpu_time_us":1518,"apply.run_wall_time_us":1517,"num_ops":2,"prepare.queue_time_us":105,"prepare.run_cpu_time_us":2030,"prepare.run_wall_time_us":6427,"raft.queue_time_us":986,"raft.run_cpu_time_us":676,"raft.run_wall_time_us":677,"replication_time_us":12809}]]}
I20250626 19:57:34.084233 5410 raft_consensus.cc:383] T 33a75d223c274c049cf25f79fcc4b961 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:34.106173 5448 raft_consensus.cc:491] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:34.112762 5448 raft_consensus.cc:513] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.114226 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet afd1fafb53434957b002fad5c77431d4 (table default.Table_b6e3c1f4224b4dee8c044d2548276942 [id=cb6e1a290c9c47c0bef9de879ab06bd9]) successfully deleted
I20250626 19:57:34.114645 5448 leader_election.cc:290] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:34.115273 5410 raft_consensus.cc:738] T 33a75d223c274c049cf25f79fcc4b961 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:34.115638 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c92deaa237064dee8f866e3a7a355ca4" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:34.115995 5410 consensus_queue.cc:260] T 33a75d223c274c049cf25f79fcc4b961 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:34.116823 4609 leader_election.cc:343] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:34.118726 5353 tablet_bootstrap.cc:492] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
W20250626 19:57:34.122071 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet df3da12971654d9abc2ee8840c4f1592 because the tablet was not found. No further retry: Not found: Tablet not found: df3da12971654d9abc2ee8840c4f1592
I20250626 19:57:34.126945 5410 ts_tablet_manager.cc:1428] T 33a75d223c274c049cf25f79fcc4b961 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.088s user 0.007s sys 0.001s
I20250626 19:57:34.127740 5433 tablet_replica.cc:331] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:34.128477 5438 ts_tablet_manager.cc:1918] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:34.133141 4975 raft_consensus.cc:1273] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 3 FOLLOWER]: Refusing update from remote peer 93f885c6802341c889a40e9c20fbfff1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250626 19:57:34.133256 4394 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_605c8b4275614ae2a45a5ad6224f3a43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:34.136548 5410 tablet_bootstrap.cc:492] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:34.139237 5438 log.cc:1199] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/1e453a4f8ca44f4a940928d9f4fcd5be
I20250626 19:57:34.140050 5438 ts_tablet_manager.cc:1939] T 1e453a4f8ca44f4a940928d9f4fcd5be P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:34.137884 5431 ts_tablet_manager.cc:1918] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:34.155510 5438 tablet_replica.cc:331] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:34.155318 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 1e453a4f8ca44f4a940928d9f4fcd5be (table default.Table_f579d2c76b104fc88466bcdcd54d2603 [id=049afd5ef383451fa96431d71b4f2ae4]) successfully deleted
I20250626 19:57:34.156664 5438 raft_consensus.cc:2241] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:34.158797 5438 raft_consensus.cc:2270] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:34.158865 5431 log.cc:1199] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/04b0f44c1b564fdf98408881678a90a1
I20250626 19:57:34.159854 5431 ts_tablet_manager.cc:1939] T 04b0f44c1b564fdf98408881678a90a1 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:34.157742 5366 tablet_bootstrap.cc:654] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:34.161443 4398 master_service.cc:948] ReplaceTablet: received request to replace tablet 1cb6bda529624ac5b5a87a03b4a02d56 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:34.164651 5438 ts_tablet_manager.cc:1905] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:34.154641 5353 tablet_bootstrap.cc:654] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:34.169767 5410 tablet_bootstrap.cc:654] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:34.175267 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30f1d01851be485f9d766784855ba627" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:34.179009 5433 raft_consensus.cc:2241] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:34.179603 5410 tablet_bootstrap.cc:492] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:34.179672 5433 raft_consensus.cc:2270] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:34.180011 5410 ts_tablet_manager.cc:1397] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.003s
I20250626 19:57:34.181692 5433 ts_tablet_manager.cc:1905] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:34.184053 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c990e4eca5cf4a8e8295e9599cf5be29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:34.193161 5448 raft_consensus.cc:491] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:34.193238 4954 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41356 (request call id 19) took 1056 ms. Trace:
I20250626 19:57:34.193635 5448 raft_consensus.cc:513] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:34.193571 4954 rpcz_store.cc:276] 0626 19:57:33.136828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.136966 (+ 138us) service_pool.cc:225] Handling call
0626 19:57:33.592314 (+455348us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0626 19:57:33.604084 (+ 11770us) ts_tablet_manager.cc:676] Creating new metadata...
0626 19:57:33.669760 (+ 65676us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:34.096548 (+426788us) tablet_replica.cc:206] Creating consensus instance
0626 19:57:34.193182 (+ 96634us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 19:57:34.195485 5448 leader_election.cc:290] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:34.197063 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 17) took 1211 ms. Trace:
I20250626 19:57:34.197445 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a31f1a7739544706bff2a500d6026086" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:34.197472 5431 rpcz_store.cc:276] 0626 19:57:32.985964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:32.990848 (+ 4884us) service_pool.cc:225] Handling call
0626 19:57:33.125982 (+135134us) ts_tablet_manager.cc:1145] Deleting tablet 04b0f44c1b564fdf98408881678a90a1
0626 19:57:33.126057 (+ 75us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:33.625830 (+499773us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b5400071554. stack: 00007f2c08de0483 00007f2c08de0380 00007f2c0895c574 00000000004fa174 00000000004fa109 00007f2c0fce2481 00007f2c0fd65f42 00007f2c1018362c 00007f2c10d96d1a 00007f2c10dc6b70 00007f2c10dadf4d 00007f2c10dadef9 00007f2c10dade89 00007f2c10dade51 00007f2c10dad07d 000000000059bcd4
0626 19:57:33.916754 (+290924us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:34.137639 (+220885us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:34.196987 (+ 59348us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":813744512,"tablet-delete.queue_time_us":134434,"thread_start_us":135990,"threads_started":1}
I20250626 19:57:34.198710 4816 raft_consensus.cc:2466] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:34.199330 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 04b0f44c1b564fdf98408881678a90a1 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]) successfully deleted
I20250626 19:57:34.199779 4609 leader_election.cc:304] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:34.200537 5448 raft_consensus.cc:2802] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:34.200874 5431 tablet_replica.cc:331] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:34.200862 5448 raft_consensus.cc:491] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:34.201164 5448 raft_consensus.cc:3058] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:34.201809 5431 raft_consensus.cc:2241] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:34.202848 5431 raft_consensus.cc:2270] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:34.207983 5448 raft_consensus.cc:513] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:34.210464 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a31f1a7739544706bff2a500d6026086" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:34.211010 4821 raft_consensus.cc:3058] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:34.211113 5448 leader_election.cc:290] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:34.212015 5410 raft_consensus.cc:357] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.212643 5410 raft_consensus.cc:383] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:34.212886 5410 raft_consensus.cc:738] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:34.213457 5410 consensus_queue.cc:260] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.220916 4821 raft_consensus.cc:2466] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:34.228091 5416 consensus_queue.cc:1035] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:34.209689 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61fcd55f6d7b4a369383d4f73d5f483d" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:34.211997 4974 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c990e4eca5cf4a8e8295e9599cf5be29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:34.229692 4609 leader_election.cc:304] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:34.230406 5368 raft_consensus.cc:2802] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:34.231150 5368 raft_consensus.cc:695] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:34.231892 5368 consensus_queue.cc:237] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:34.220980 5483 tablet.cc:1722] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250626 19:57:34.228988 5353 tablet_bootstrap.cc:492] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:34.236460 5353 ts_tablet_manager.cc:1397] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.131s user 0.013s sys 0.000s
W20250626 19:57:34.237337 4611 leader_election.cc:343] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:34.238152 4611 leader_election.cc:343] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:34.238931 4611 leader_election.cc:343] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:34.239778 4611 leader_election.cc:343] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:34.240023 5431 ts_tablet_manager.cc:1905] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:34.240149 4611 leader_election.cc:304] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:34.246022 5444 raft_consensus.cc:2747] T 30f1d01851be485f9d766784855ba627 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:34.248003 4611 leader_election.cc:343] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:34.257822 4756 leader_election.cc:343] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:34.258975 4756 leader_election.cc:343] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:34.259344 4756 leader_election.cc:304] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:34.260353 5429 raft_consensus.cc:2747] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:34.260221 4611 leader_election.cc:304] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:34.260932 5444 raft_consensus.cc:2747] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:57:34.262791 5429 consensus_queue.cc:1035] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:34.263077 4673 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fda659c30c44c8ba402ac5c7f3598e0" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:34.264662 4756 leader_election.cc:343] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:34.238565 5353 raft_consensus.cc:357] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:34.276264 5353 raft_consensus.cc:383] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:34.276618 5353 raft_consensus.cc:738] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:34.277257 5353 consensus_queue.cc:260] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:34.281806 5353 ts_tablet_manager.cc:1428] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20250626 19:57:34.282150 5366 tablet_bootstrap.cc:492] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
W20250626 19:57:34.282495 4756 leader_election.cc:343] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:34.285795 5433 ts_tablet_manager.cc:1918] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250626 19:57:34.285800 4756 consensus_peers.cc:487] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:34.286476 5416 consensus_queue.cc:1035] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:57:34.288087 4756 leader_election.cc:343] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:34.288506 4756 leader_election.cc:304] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:34.289158 5416 raft_consensus.cc:2747] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:34.290500 4756 leader_election.cc:343] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:34.290942 4756 leader_election.cc:304] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:34.291590 5416 raft_consensus.cc:2747] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:57:34.294574 5366 ts_tablet_manager.cc:1397] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.195s user 0.010s sys 0.006s
I20250626 19:57:34.300884 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 94cefa59de044a36a2afaaf5531128d8 (DEFAULT_TABLE table=default.Table_c580e8a6ee5d417caa2f7d6811a68349 [id=b8bf68764b2c4a94b7299fa273983e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.301864 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94cefa59de044a36a2afaaf5531128d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.311794 4798 tablet_service.cc:1468] Processing CreateTablet for tablet 4fb3692f4ddb4ff89c1817da4a5cb927 (DEFAULT_TABLE table=default.Table_7a020d26ba3141acaca632ae611a203b [id=c0885b872a864765acb92aca10e1bd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.312130 5353 tablet_bootstrap.cc:492] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:34.312747 4798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fb3692f4ddb4ff89c1817da4a5cb927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.298040 4957 tablet_service.cc:1468] Processing CreateTablet for tablet 94cefa59de044a36a2afaaf5531128d8 (DEFAULT_TABLE table=default.Table_c580e8a6ee5d417caa2f7d6811a68349 [id=b8bf68764b2c4a94b7299fa273983e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.315439 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94cefa59de044a36a2afaaf5531128d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.302691 4956 tablet_service.cc:1468] Processing CreateTablet for tablet 4fb3692f4ddb4ff89c1817da4a5cb927 (DEFAULT_TABLE table=default.Table_7a020d26ba3141acaca632ae611a203b [id=c0885b872a864765acb92aca10e1bd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.319484 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fb3692f4ddb4ff89c1817da4a5cb927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.298472 4653 tablet_service.cc:1468] Processing CreateTablet for tablet 94cefa59de044a36a2afaaf5531128d8 (DEFAULT_TABLE table=default.Table_c580e8a6ee5d417caa2f7d6811a68349 [id=b8bf68764b2c4a94b7299fa273983e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.302446 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 4fb3692f4ddb4ff89c1817da4a5cb927 (DEFAULT_TABLE table=default.Table_7a020d26ba3141acaca632ae611a203b [id=c0885b872a864765acb92aca10e1bd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.305315 4912 leader_election.cc:304] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:34.327555 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fb3692f4ddb4ff89c1817da4a5cb927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.327199 5366 raft_consensus.cc:357] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.327847 5366 raft_consensus.cc:383] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:34.329130 5410 ts_tablet_manager.cc:1428] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.149s user 0.007s sys 0.000s
I20250626 19:57:34.332083 5366 raft_consensus.cc:738] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:34.330199 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94cefa59de044a36a2afaaf5531128d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.335723 5410 tablet_bootstrap.cc:492] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:34.336320 5407 raft_consensus.cc:2747] T 9f91a78f6d474133976242ea8d259e29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 19:57:34.342356 4755 consensus_peers.cc:487] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:34.342285 4912 leader_election.cc:304] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:34.343734 5255 raft_consensus.cc:3058] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:34.343071 5455 consensus_queue.cc:1035] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:34.334410 5366 consensus_queue.cc:260] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.334856 5433 log.cc:1199] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/5785508e37624f02bc2f6cc93007e3f8
I20250626 19:57:34.345608 5433 ts_tablet_manager.cc:1939] T 5785508e37624f02bc2f6cc93007e3f8 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:34.330420 5448 raft_consensus.cc:491] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:34.352221 5255 raft_consensus.cc:2747] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:34.359150 4954 tablet_service.cc:1468] Processing CreateTablet for tablet e60b4d53c40c4dffab928f0e3fb4a80b (DEFAULT_TABLE table=default.Table_e25c8c1da59041118a0072aa639e3771 [id=bea62f96f1cd4c2ab35704acf7def541]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.360718 5429 raft_consensus.cc:491] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:34.360872 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e60b4d53c40c4dffab928f0e3fb4a80b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.361184 5429 raft_consensus.cc:513] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.362773 5433 tablet_replica.cc:331] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:34.363164 5429 leader_election.cc:290] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:34.363623 5433 raft_consensus.cc:2241] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:34.364097 5433 raft_consensus.cc:2270] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:34.363920 4651 tablet_service.cc:1468] Processing CreateTablet for tablet e60b4d53c40c4dffab928f0e3fb4a80b (DEFAULT_TABLE table=default.Table_e25c8c1da59041118a0072aa639e3771 [id=bea62f96f1cd4c2ab35704acf7def541]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.365594 4651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e60b4d53c40c4dffab928f0e3fb4a80b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.366472 5433 ts_tablet_manager.cc:1905] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:34.367219 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 5785508e37624f02bc2f6cc93007e3f8 (table default.Table_b401c8a5ab6744029f693b1b5e0f306d [id=95832380520e4cf18d762654fb684db3]) successfully deleted
I20250626 19:57:34.376523 4798 tablet_service.cc:1468] Processing CreateTablet for tablet e60b4d53c40c4dffab928f0e3fb4a80b (DEFAULT_TABLE table=default.Table_e25c8c1da59041118a0072aa639e3771 [id=bea62f96f1cd4c2ab35704acf7def541]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.377562 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_ef61f7b59da5409ca6ba9ab8d9ecf944"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:34.379209 4798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e60b4d53c40c4dffab928f0e3fb4a80b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:34.379639 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef61f7b59da5409ca6ba9ab8d9ecf944 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:34.380517 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33a75d223c274c049cf25f79fcc4b961" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:34.380493 4974 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fda659c30c44c8ba402ac5c7f3598e0" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:34.380613 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c92deaa237064dee8f866e3a7a355ca4" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:34.381974 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a31f1a7739544706bff2a500d6026086" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:34.383428 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a31f1a7739544706bff2a500d6026086" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
W20250626 19:57:34.385138 4611 leader_election.cc:343] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:34.387506 4394 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_a55a71930b3e4026aaa4b8515291c5f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:34.387717 4611 leader_election.cc:304] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
W20250626 19:57:34.389197 4394 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a55a71930b3e4026aaa4b8515291c5f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:34.390682 5429 raft_consensus.cc:491] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:34.391218 5429 raft_consensus.cc:513] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.394248 5429 leader_election.cc:290] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:34.390707 4974 raft_consensus.cc:2466] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:34.400173 4676 raft_consensus.cc:2466] T 33a75d223c274c049cf25f79fcc4b961 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
W20250626 19:57:34.401496 4611 leader_election.cc:343] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:34.404801 4611 leader_election.cc:343] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:34.399562 5366 ts_tablet_manager.cc:1428] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.104s user 0.003s sys 0.002s
I20250626 19:57:34.414695 5366 tablet_bootstrap.cc:492] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:34.426889 5353 tablet_bootstrap.cc:654] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:34.430147 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43aed8f07844c5ca727ae6df63896df" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:34.434903 4974 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33a75d223c274c049cf25f79fcc4b961" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:34.435905 4755 leader_election.cc:304] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:34.436759 5360 raft_consensus.cc:2802] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:34.437152 5360 raft_consensus.cc:491] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:34.437549 5360 raft_consensus.cc:3058] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:34.399711 5368 raft_consensus.cc:2747] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:34.442529 5438 ts_tablet_manager.cc:1918] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250626 19:57:34.443323 4756 leader_election.cc:343] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:34.447646 4756 leader_election.cc:343] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:34.443045 5438 log.cc:1199] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/4d8ffa9b658c413fa06d02ee6635dd1b
I20250626 19:57:34.451615 5438 ts_tablet_manager.cc:1939] T 4d8ffa9b658c413fa06d02ee6635dd1b P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:34.454084 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 4d8ffa9b658c413fa06d02ee6635dd1b (table default.Table_681222687426414bb0838e947d315979 [id=4419bc765f3a4ff1a104bb656bf642ea]) successfully deleted
I20250626 19:57:34.455572 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_8ed57e01e9b54a829b11bc477378d778"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:34.457271 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ed57e01e9b54a829b11bc477378d778 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:34.461020 5438 tablet_replica.cc:331] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:34.472414 5366 tablet_bootstrap.cc:654] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:34.457479 5360 raft_consensus.cc:513] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.474177 5438 raft_consensus.cc:2241] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:34.399708 5448 raft_consensus.cc:513] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.476598 5438 raft_consensus.cc:2270] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:34.479879 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 44960f8db08942879cde6f96e9c620da (table default.Table_eff15db1de2e4aa3977a66f4af2b3dcd [id=8df233b9523c49adb3d24727bda24b9e]). Expected version 2 got 0
I20250626 19:57:34.480393 5360 leader_election.cc:290] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:34.480762 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet a3b05549a1434a1fa3d337fdfb96a131 (table default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]). Expected version 1 got 0
I20250626 19:57:34.481359 5438 ts_tablet_manager.cc:1905] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:34.482509 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43aed8f07844c5ca727ae6df63896df" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:34.483137 4674 raft_consensus.cc:2466] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:34.481904 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 1cb6bda529624ac5b5a87a03b4a02d56 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:34.484496 5448 leader_election.cc:290] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:34.487125 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33a75d223c274c049cf25f79fcc4b961" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:34.487296 4755 leader_election.cc:304] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:34.487660 4674 raft_consensus.cc:3058] T 33a75d223c274c049cf25f79fcc4b961 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:34.488520 5429 raft_consensus.cc:2802] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:34.488827 5429 raft_consensus.cc:491] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:34.489149 5429 raft_consensus.cc:3058] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:34.490639 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7facb77eb579448f9533d285cba8650e" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
W20250626 19:57:34.491804 4609 leader_election.cc:343] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:34.493016 4394 master_service.cc:948] ReplaceTablet: received request to replace tablet fd6543699e254400b3516aece4486fe0 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:34.502609 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:34.503373 5366 tablet_bootstrap.cc:492] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:34.503849 5366 ts_tablet_manager.cc:1397] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.089s user 0.008s sys 0.002s
I20250626 19:57:34.503734 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 5e20ea7c17c34fbd9c117275e3c14466 (table default.Table_84725c81a9a9432c8232f87fb7627f6a [id=089059dab32a41179048a6eb4473a99a]). Expected version 1 got 0
I20250626 19:57:34.509292 5366 raft_consensus.cc:357] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.510027 5366 raft_consensus.cc:383] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:34.510337 5366 raft_consensus.cc:738] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:34.512676 5410 tablet_bootstrap.cc:654] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:34.498359 5429 raft_consensus.cc:513] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.517148 5366 consensus_queue.cc:260] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.519774 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43aed8f07844c5ca727ae6df63896df" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:34.517863 5429 leader_election.cc:290] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:34.520325 4677 raft_consensus.cc:3058] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:34.531709 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_f93f48d328f04df8b27e717e9481fd8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:34.533411 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f93f48d328f04df8b27e717e9481fd8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:34.535036 4394 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet fd6543699e254400b3516aece4486fe0
I20250626 19:57:34.536549 4394 catalog_manager.cc:6652] ReplaceTablet: tablet fd6543699e254400b3516aece4486fe0 deleted and replaced by tablet 1474bfa74a6346bf9e3ff49019f6c9b3
I20250626 19:57:34.539865 4798 tablet_service.cc:1515] Processing DeleteTablet for tablet fd6543699e254400b3516aece4486fe0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1474bfa74a6346bf9e3ff49019f6c9b3) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:34.543078 4677 raft_consensus.cc:2466] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:34.543226 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet fd6543699e254400b3516aece4486fe0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1474bfa74a6346bf9e3ff49019f6c9b3) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:34.544078 4755 leader_election.cc:304] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:34.545099 5366 ts_tablet_manager.cc:1428] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.041s user 0.004s sys 0.002s
I20250626 19:57:34.545922 5366 tablet_bootstrap.cc:492] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:34.495847 4674 raft_consensus.cc:2466] T 33a75d223c274c049cf25f79fcc4b961 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:34.547523 5429 raft_consensus.cc:2802] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:34.548182 5429 raft_consensus.cc:695] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:34.554883 5366 tablet_bootstrap.cc:654] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:34.549576 5497 tablet.cc:1722] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:34.560642 5429 consensus_queue.cc:237] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.564141 4755 leader_election.cc:304] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:34.565289 5360 raft_consensus.cc:2802] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:34.565790 5360 raft_consensus.cc:695] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:34.566730 5360 consensus_queue.cc:237] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.576572 5438 ts_tablet_manager.cc:1918] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:34.531849 5255 raft_consensus.cc:491] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:34.581903 5410 tablet_bootstrap.cc:492] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:34.586144 5410 ts_tablet_manager.cc:1397] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.253s user 0.013s sys 0.002s
I20250626 19:57:34.591437 5438 log.cc:1199] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/ea7374ad6ba64e46b7df9852fa33f145
I20250626 19:57:34.593114 5410 raft_consensus.cc:357] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.593814 5410 raft_consensus.cc:383] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:34.594105 5410 raft_consensus.cc:738] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:34.595582 5438 ts_tablet_manager.cc:1939] T ea7374ad6ba64e46b7df9852fa33f145 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:34.603677 5366 tablet_bootstrap.cc:492] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:34.607192 5366 ts_tablet_manager.cc:1397] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.004s
I20250626 19:57:34.609815 5366 raft_consensus.cc:357] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.611053 5410 consensus_queue.cc:260] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.613065 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet ea7374ad6ba64e46b7df9852fa33f145 (table default.Table_3f8afab3426b4088b717c05b65bb84e5 [id=ca71ef30f2ff4908b3b558d31e1cf989]) successfully deleted
I20250626 19:57:34.620251 5410 ts_tablet_manager.cc:1428] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.029s user 0.004s sys 0.001s
I20250626 19:57:34.621234 5366 raft_consensus.cc:383] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:34.621960 5366 raft_consensus.cc:738] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:34.614326 5438 tablet_replica.cc:331] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:34.623590 5438 raft_consensus.cc:2241] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:34.624523 5438 raft_consensus.cc:2270] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:34.626772 5438 ts_tablet_manager.cc:1905] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:34.627701 5433 ts_tablet_manager.cc:1918] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:34.628157 5433 log.cc:1199] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/78a8f494f8614e7ebb8ca880b6c80e78
I20250626 19:57:34.628958 5433 ts_tablet_manager.cc:1939] T 78a8f494f8614e7ebb8ca880b6c80e78 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:34.629973 5410 tablet_bootstrap.cc:492] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
W20250626 19:57:34.635099 4383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20250626 19:57:34.634992 5504 tablet.cc:1722] T 44960f8db08942879cde6f96e9c620da P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250626 19:57:34.643101 4974 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43aed8f07844c5ca727ae6df63896df" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:34.644217 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33a75d223c274c049cf25f79fcc4b961" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:34.645067 5255 raft_consensus.cc:513] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:34.645991 5410 tablet_bootstrap.cc:654] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:34.647310 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7facb77eb579448f9533d285cba8650e" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:34.627348 5366 consensus_queue.cc:260] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.653650 5433 tablet_replica.cc:331] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:34.655395 5366 ts_tablet_manager.cc:1428] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.047s user 0.005s sys 0.000s
I20250626 19:57:34.656210 5366 tablet_bootstrap.cc:492] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
W20250626 19:57:34.666467 4383 consensus_peers.cc:487] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af -> Peer e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823): Couldn't send request to peer e6c58006b6be41c1bd6bdb5eb010d360. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250626 19:57:34.667704 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_c010e84626544214a45a911f02c4d903"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:34.667915 4394 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_34846ed4a33e42aa8811e45b8d029a8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:34.670030 4394 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34846ed4a33e42aa8811e45b8d029a8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:34.654677 5433 raft_consensus.cc:2241] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:34.676635 5433 raft_consensus.cc:2270] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:34.679872 5433 ts_tablet_manager.cc:1905] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:34.680655 5366 tablet_bootstrap.cc:654] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:34.683887 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c010e84626544214a45a911f02c4d903 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:34.703379 5366 tablet_bootstrap.cc:492] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:34.703758 5366 ts_tablet_manager.cc:1397] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.000s
I20250626 19:57:34.705888 5366 raft_consensus.cc:357] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.706480 5366 raft_consensus.cc:383] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:34.706701 5366 raft_consensus.cc:738] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:34.711642 5366 consensus_queue.cc:260] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.702507 5438 ts_tablet_manager.cc:1918] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:34.712911 5438 log.cc:1199] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/afd1fafb53434957b002fad5c77431d4
I20250626 19:57:34.714103 5438 ts_tablet_manager.cc:1939] T afd1fafb53434957b002fad5c77431d4 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:34.719836 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 30) took 1015 ms. Trace:
I20250626 19:57:34.724218 5438 rpcz_store.cc:276] 0626 19:57:33.704842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.704963 (+ 121us) service_pool.cc:225] Handling call
0626 19:57:34.614166 (+909203us) ts_tablet_manager.cc:1145] Deleting tablet afd1fafb53434957b002fad5c77431d4
0626 19:57:34.614210 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:34.685589 (+ 71379us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:34.702268 (+ 16679us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:34.719779 (+ 17511us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":908655}
I20250626 19:57:34.725308 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 78a8f494f8614e7ebb8ca880b6c80e78 (table default.Table_cc421ddfd27f48b2a8a3bf433f8720f8 [id=eec0405794114b30877bb326de9f9076]) successfully deleted
I20250626 19:57:34.726848 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet afd1fafb53434957b002fad5c77431d4 (table default.Table_b6e3c1f4224b4dee8c044d2548276942 [id=cb6e1a290c9c47c0bef9de879ab06bd9]) successfully deleted
I20250626 19:57:34.721237 5366 ts_tablet_manager.cc:1428] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250626 19:57:34.727187 5433 ts_tablet_manager.cc:1918] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:34.730293 5366 tablet_bootstrap.cc:492] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:34.735365 5438 tablet_replica.cc:331] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:34.736778 5438 raft_consensus.cc:2241] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:34.738257 5438 raft_consensus.cc:2270] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:34.744083 5433 log.cc:1199] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/ea7374ad6ba64e46b7df9852fa33f145
I20250626 19:57:34.745107 5438 ts_tablet_manager.cc:1905] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:34.746059 5433 ts_tablet_manager.cc:1939] T ea7374ad6ba64e46b7df9852fa33f145 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:34.747413 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 30) took 1020 ms. Trace:
I20250626 19:57:34.748991 5433 rpcz_store.cc:276] 0626 19:57:33.726941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.730660 (+ 3719us) service_pool.cc:225] Handling call
0626 19:57:34.653455 (+922795us) ts_tablet_manager.cc:1145] Deleting tablet ea7374ad6ba64e46b7df9852fa33f145
0626 19:57:34.653509 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:34.710200 (+ 56691us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:34.726907 (+ 16707us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:34.747348 (+ 20441us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":868918}
I20250626 19:57:34.751391 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_822bad36b81c42ac9101bcab4aaca9a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:34.754892 5433 tablet_replica.cc:331] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:34.755810 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:34.758136 5433 raft_consensus.cc:2241] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:34.760224 5366 tablet_bootstrap.cc:654] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:34.763309 5433 raft_consensus.cc:2270] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:34.751448 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet ea7374ad6ba64e46b7df9852fa33f145 (table default.Table_3f8afab3426b4088b717c05b65bb84e5 [id=ca71ef30f2ff4908b3b558d31e1cf989]) successfully deleted
W20250626 19:57:34.772753 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_822bad36b81c42ac9101bcab4aaca9a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:34.773857 5433 ts_tablet_manager.cc:1905] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:34.774016 5366 tablet_bootstrap.cc:492] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:34.774448 5366 ts_tablet_manager.cc:1397] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.046s user 0.012s sys 0.001s
I20250626 19:57:34.776752 5366 raft_consensus.cc:357] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.785013 5366 raft_consensus.cc:383] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:34.785296 5366 raft_consensus.cc:738] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:34.790248 5366 consensus_queue.cc:260] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:34.812901 4394 master_service.cc:948] ReplaceTablet: received request to replace tablet 4b3bed19e7e34d24b995840a3bde5ef0 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:34.827646 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:34.855970 4394 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 4b3bed19e7e34d24b995840a3bde5ef0
I20250626 19:57:34.857775 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b3bed19e7e34d24b995840a3bde5ef0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c113e03594b428a979435618fe373c6) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:34.859624 4394 catalog_manager.cc:6652] ReplaceTablet: tablet 4b3bed19e7e34d24b995840a3bde5ef0 deleted and replaced by tablet 5c113e03594b428a979435618fe373c6
I20250626 19:57:34.862582 5366 ts_tablet_manager.cc:1428] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.088s user 0.007s sys 0.002s
I20250626 19:57:34.863642 5366 tablet_bootstrap.cc:492] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:34.863785 5416 consensus_queue.cc:1035] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:57:34.867182 4756 leader_election.cc:343] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:34.868541 4756 leader_election.cc:343] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:34.878114 4912 consensus_peers.cc:487] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78a8f494f8614e7ebb8ca880b6c80e78. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:34.880960 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e20ea7c17c34fbd9c117275e3c14466" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:34.881711 4816 raft_consensus.cc:2391] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:34.881170 4395 catalog_manager.cc:5582] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:34.883031 4395 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 44960f8db08942879cde6f96e9c620da (table default.Table_eff15db1de2e4aa3977a66f4af2b3dcd [id=8df233b9523c49adb3d24727bda24b9e]). Expected version 2 got 0
I20250626 19:57:34.887209 4798 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b3bed19e7e34d24b995840a3bde5ef0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c113e03594b428a979435618fe373c6) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:34.887303 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 0f834922c9de46dca48c47748913b11a (DEFAULT_TABLE table=default.Table_89c3869cb74248c4af47e8eaa3371b16 [id=594e3d715a6d42c798c44d20740d2ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.888475 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f834922c9de46dca48c47748913b11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.889227 5438 ts_tablet_manager.cc:1918] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:34.893136 5368 consensus_queue.cc:1035] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:34.890185 4798 tablet_service.cc:1468] Processing CreateTablet for tablet 0f834922c9de46dca48c47748913b11a (DEFAULT_TABLE table=default.Table_89c3869cb74248c4af47e8eaa3371b16 [id=594e3d715a6d42c798c44d20740d2ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.896767 4798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f834922c9de46dca48c47748913b11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.896656 4653 tablet_service.cc:1468] Processing CreateTablet for tablet 419c4018256546ef9431ac2db619ee33 (DEFAULT_TABLE table=default.Table_dafe34566c694a47a3d9321791db3fb8 [id=7f7e05531925475d937c9035f882dab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.897691 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 419c4018256546ef9431ac2db619ee33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.898412 4395 catalog_manager.cc:5582] T 9f91a78f6d474133976242ea8d259e29 P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:34.885363 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_428898f0f296481cb11e678bc15af894"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:34.901736 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_428898f0f296481cb11e678bc15af894 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:34.904212 4651 tablet_service.cc:1468] Processing CreateTablet for tablet 6c86ced5a3dc48e380244a9548d2f4c8 (DEFAULT_TABLE table=default.Table_c60978a633d54c628ce6fbb9e0cdac48 [id=b1a0ecc621c149af94b56f61f584101e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.905269 4651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c86ced5a3dc48e380244a9548d2f4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.906914 5438 log.cc:1199] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/512dc911bfce49b29c6fc8cd90523723
I20250626 19:57:34.907537 4652 tablet_service.cc:1468] Processing CreateTablet for tablet 6b8473259485460783f62f8193d5b02c (DEFAULT_TABLE table=default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.908540 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b8473259485460783f62f8193d5b02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.907689 5438 ts_tablet_manager.cc:1939] T 512dc911bfce49b29c6fc8cd90523723 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:34.917936 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 32) took 1212 ms. Trace:
I20250626 19:57:34.918269 5438 rpcz_store.cc:276] 0626 19:57:33.705414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.705526 (+ 112us) service_pool.cc:225] Handling call
0626 19:57:34.735206 (+1029680us) ts_tablet_manager.cc:1145] Deleting tablet 512dc911bfce49b29c6fc8cd90523723
0626 19:57:34.735252 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:34.779096 (+ 43844us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:34.888990 (+109894us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:34.917881 (+ 28891us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1029132}
I20250626 19:57:34.922425 5368 consensus_queue.cc:1035] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:34.923395 4956 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b3bed19e7e34d24b995840a3bde5ef0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c113e03594b428a979435618fe373c6) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:34.922236 4395 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet a3b05549a1434a1fa3d337fdfb96a131 (table default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]). Expected version 1 got 0
I20250626 19:57:34.879182 5255 leader_election.cc:290] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
W20250626 19:57:34.885681 4911 consensus_peers.cc:487] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Couldn't send request to peer 93f885c6802341c889a40e9c20fbfff1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea7374ad6ba64e46b7df9852fa33f145. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:34.927268 4958 tablet_service.cc:1468] Processing CreateTablet for tablet 6c86ced5a3dc48e380244a9548d2f4c8 (DEFAULT_TABLE table=default.Table_c60978a633d54c628ce6fbb9e0cdac48 [id=b1a0ecc621c149af94b56f61f584101e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.928246 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c86ced5a3dc48e380244a9548d2f4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.929973 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e20ea7c17c34fbd9c117275e3c14466" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:34.932793 4611 leader_election.cc:343] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:34.933197 4611 leader_election.cc:304] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:34.934145 5368 raft_consensus.cc:2747] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:34.939356 4654 tablet_service.cc:1468] Processing CreateTablet for tablet 82823fd9ceb1485386616319fa7aeb43 (DEFAULT_TABLE table=default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.940284 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82823fd9ceb1485386616319fa7aeb43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.944414 5410 tablet_bootstrap.cc:492] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:34.944816 5410 ts_tablet_manager.cc:1397] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.315s user 0.012s sys 0.000s
I20250626 19:57:34.947891 4956 tablet_service.cc:1468] Processing CreateTablet for tablet 0f834922c9de46dca48c47748913b11a (DEFAULT_TABLE table=default.Table_89c3869cb74248c4af47e8eaa3371b16 [id=594e3d715a6d42c798c44d20740d2ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.948818 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f834922c9de46dca48c47748913b11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.949779 5433 ts_tablet_manager.cc:1918] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:34.950112 5433 log.cc:1199] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/af58f34e2248432fae5c084bd3388028
I20250626 19:57:34.950735 5433 ts_tablet_manager.cc:1939] T af58f34e2248432fae5c084bd3388028 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
W20250626 19:57:34.957979 4609 consensus_peers.cc:487] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 -> Peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Couldn't send request to peer 93f885c6802341c889a40e9c20fbfff1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 512dc911bfce49b29c6fc8cd90523723. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:34.961249 4395 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 5e20ea7c17c34fbd9c117275e3c14466 (table default.Table_84725c81a9a9432c8232f87fb7627f6a [id=089059dab32a41179048a6eb4473a99a]). Expected version 1 got 0
I20250626 19:57:34.929508 5366 tablet_bootstrap.cc:654] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:34.931531 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 419c4018256546ef9431ac2db619ee33 (DEFAULT_TABLE table=default.Table_dafe34566c694a47a3d9321791db3fb8 [id=7f7e05531925475d937c9035f882dab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.968824 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 419c4018256546ef9431ac2db619ee33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.937309 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 6c86ced5a3dc48e380244a9548d2f4c8 (DEFAULT_TABLE table=default.Table_c60978a633d54c628ce6fbb9e0cdac48 [id=b1a0ecc621c149af94b56f61f584101e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.971341 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c86ced5a3dc48e380244a9548d2f4c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:34.955650 4756 consensus_peers.cc:487] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:34.956272 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 6b8473259485460783f62f8193d5b02c (DEFAULT_TABLE table=default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.974013 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b8473259485460783f62f8193d5b02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.962795 4795 tablet_service.cc:1468] Processing CreateTablet for tablet 82823fd9ceb1485386616319fa7aeb43 (DEFAULT_TABLE table=default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.975350 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82823fd9ceb1485386616319fa7aeb43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:34.975490 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_ff801b0f32eb4c948c82f6c4b7d42b97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:34.979039 4384 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 5e20ea7c17c34fbd9c117275e3c14466 (table default.Table_84725c81a9a9432c8232f87fb7627f6a [id=089059dab32a41179048a6eb4473a99a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5e20ea7c17c34fbd9c117275e3c14466 consensus state has no leader
I20250626 19:57:34.976882 4395 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 1cb6bda529624ac5b5a87a03b4a02d56 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:34.983462 4957 tablet_service.cc:1468] Processing CreateTablet for tablet 419c4018256546ef9431ac2db619ee33 (DEFAULT_TABLE table=default.Table_dafe34566c694a47a3d9321791db3fb8 [id=7f7e05531925475d937c9035f882dab7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250626 19:57:34.984673 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff801b0f32eb4c948c82f6c4b7d42b97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:34.986363 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 6b8473259485460783f62f8193d5b02c (DEFAULT_TABLE table=default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.983395 4395 catalog_manager.cc:5582] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:34.991165 4395 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:34.994141 4950 tablet_service.cc:1468] Processing CreateTablet for tablet 82823fd9ceb1485386616319fa7aeb43 (DEFAULT_TABLE table=default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:34.997084 5366 tablet_bootstrap.cc:492] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:34.997479 5366 ts_tablet_manager.cc:1397] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.134s user 0.010s sys 0.004s
I20250626 19:57:34.998803 4912 leader_election.cc:304] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:34.890049 5431 ts_tablet_manager.cc:1918] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:35.002242 5431 log.cc:1199] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/1e453a4f8ca44f4a940928d9f4fcd5be
I20250626 19:57:35.003130 5431 ts_tablet_manager.cc:1939] T 1e453a4f8ca44f4a940928d9f4fcd5be P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:35.003098 5366 raft_consensus.cc:357] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.003683 5366 raft_consensus.cc:383] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:34.901744 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet fd6543699e254400b3516aece4486fe0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1474bfa74a6346bf9e3ff49019f6c9b3) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:35.003923 5366 raft_consensus.cc:738] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:34.914868 4973 raft_consensus.cc:3058] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:35.004462 5366 consensus_queue.cc:260] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.008388 5366 ts_tablet_manager.cc:1428] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20250626 19:57:35.009224 5366 tablet_bootstrap.cc:492] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:35.022805 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 34) took 1205 ms. Trace:
I20250626 19:57:35.023151 5433 rpcz_store.cc:276] 0626 19:57:33.817786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.817907 (+ 121us) service_pool.cc:225] Handling call
0626 19:57:34.753608 (+935701us) ts_tablet_manager.cc:1145] Deleting tablet af58f34e2248432fae5c084bd3388028
0626 19:57:34.753906 (+ 298us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:34.846128 (+ 92222us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:34.949584 (+103456us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:35.022706 (+ 73122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":934619}
I20250626 19:57:35.024924 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet af58f34e2248432fae5c084bd3388028 (table default.Table_e25c8c1da59041118a0072aa639e3771 [id=bea62f96f1cd4c2ab35704acf7def541]) successfully deleted
I20250626 19:57:35.049892 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 28) took 1275 ms. Trace:
W20250626 19:57:35.051189 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 52486.6us
I20250626 19:57:35.060526 5353 tablet_bootstrap.cc:492] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:35.060995 5353 ts_tablet_manager.cc:1397] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.749s user 0.011s sys 0.010s
I20250626 19:57:35.076284 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 419c4018256546ef9431ac2db619ee33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.060056 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 512dc911bfce49b29c6fc8cd90523723 (table default.Table_89c4b3541e7c4b08b25f28abf471d42d [id=f0d69891ab4745dbb68ea3c529d1a107]) successfully deleted
I20250626 19:57:35.079285 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b8473259485460783f62f8193d5b02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.083029 4950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82823fd9ceb1485386616319fa7aeb43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.085951 5455 raft_consensus.cc:2747] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:35.087531 5410 raft_consensus.cc:357] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.087877 4398 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 1cb6bda529624ac5b5a87a03b4a02d56
I20250626 19:57:35.088274 5410 raft_consensus.cc:383] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:35.089097 4398 catalog_manager.cc:6652] ReplaceTablet: tablet 1cb6bda529624ac5b5a87a03b4a02d56 deleted and replaced by tablet 465ed0fe56fb4960a58e92b9b3d594e6
I20250626 19:57:35.088617 5410 raft_consensus.cc:738] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:35.090085 5410 consensus_queue.cc:260] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.091822 5438 tablet_replica.cc:331] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
W20250626 19:57:35.075744 4755 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 51968.1us
I20250626 19:57:35.090066 4398 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_bca98e7f86544ed89dbd4af5356026ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:35.102421 4398 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bca98e7f86544ed89dbd4af5356026ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:35.104151 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 1cb6bda529624ac5b5a87a03b4a02d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 465ed0fe56fb4960a58e92b9b3d594e6) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:35.046734 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_a5a4503fbcda4a7f95c722aaa5b4b0ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:35.121244 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:35.123665 5353 raft_consensus.cc:357] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:35.124293 5353 raft_consensus.cc:383] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:35.124539 5353 raft_consensus.cc:738] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:35.125047 5353 consensus_queue.cc:260] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:35.138392 4796 tablet_service.cc:1515] Processing DeleteTablet for tablet 1cb6bda529624ac5b5a87a03b4a02d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 465ed0fe56fb4960a58e92b9b3d594e6) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:35.139086 5506 raft_consensus.cc:491] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:35.139487 5506 raft_consensus.cc:513] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.149405 5410 ts_tablet_manager.cc:1428] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.204s user 0.006s sys 0.000s
I20250626 19:57:35.150491 5410 tablet_bootstrap.cc:492] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:35.096963 5433 tablet_replica.cc:331] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:35.151587 5433 raft_consensus.cc:2241] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:35.152638 5433 raft_consensus.cc:2270] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:35.050256 5431 rpcz_store.cc:276] 0626 19:57:33.774203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.774353 (+ 150us) service_pool.cc:225] Handling call
0626 19:57:34.200642 (+426289us) ts_tablet_manager.cc:1145] Deleting tablet 1e453a4f8ca44f4a940928d9f4fcd5be
0626 19:57:34.200697 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:34.349557 (+148860us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:34.889826 (+540269us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:35.049819 (+159993us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":425494}
I20250626 19:57:35.173957 5431 tablet_replica.cc:331] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:35.181649 5353 ts_tablet_manager.cc:1428] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.120s user 0.005s sys 0.001s
W20250626 19:57:35.122723 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5a4503fbcda4a7f95c722aaa5b4b0ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:35.182662 5353 tablet_bootstrap.cc:492] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:35.183986 5431 raft_consensus.cc:2241] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:35.163261 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 1e453a4f8ca44f4a940928d9f4fcd5be (table default.Table_f579d2c76b104fc88466bcdcd54d2603 [id=049afd5ef383451fa96431d71b4f2ae4]) successfully deleted
I20250626 19:57:35.107836 4654 tablet_service.cc:1515] Processing DeleteTablet for tablet 1cb6bda529624ac5b5a87a03b4a02d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 465ed0fe56fb4960a58e92b9b3d594e6) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:35.113003 5368 consensus_queue.cc:1035] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:35.194164 5448 consensus_queue.cc:1035] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:35.196105 5431 raft_consensus.cc:2270] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:35.199211 5431 ts_tablet_manager.cc:1905] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:35.176306 4912 consensus_peers.cc:487] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea7374ad6ba64e46b7df9852fa33f145. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:35.205624 5433 ts_tablet_manager.cc:1905] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:35.108387 5429 raft_consensus.cc:491] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:35.239032 5429 raft_consensus.cc:513] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.243542 5410 tablet_bootstrap.cc:654] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:35.250427 5448 raft_consensus.cc:491] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:35.251005 5448 raft_consensus.cc:513] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.253185 5448 leader_election.cc:290] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:35.263422 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_07d4c194b22c4529b42fc16c2ebaff31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:35.266369 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07d4c194b22c4529b42fc16c2ebaff31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:35.276829 4977 raft_consensus.cc:3058] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:35.283849 5353 tablet_bootstrap.cc:654] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:35.292887 4611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 137635us
I20250626 19:57:35.300429 5410 tablet_bootstrap.cc:492] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:35.301005 5410 ts_tablet_manager.cc:1397] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.151s user 0.011s sys 0.001s
I20250626 19:57:35.303587 5410 raft_consensus.cc:357] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.304288 5410 raft_consensus.cc:383] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:35.304580 5410 raft_consensus.cc:738] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
W20250626 19:57:35.305261 4611 consensus_peers.cc:487] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:35.306573 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c0338c05a974e0f96865398722c5e8f" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:35.316344 5353 tablet_bootstrap.cc:492] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:35.317790 5433 ts_tablet_manager.cc:1918] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:35.319494 5353 ts_tablet_manager.cc:1397] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.137s user 0.008s sys 0.003s
I20250626 19:57:35.320290 5433 log.cc:1199] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/512dc911bfce49b29c6fc8cd90523723
I20250626 19:57:35.321000 5433 ts_tablet_manager.cc:1939] T 512dc911bfce49b29c6fc8cd90523723 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:35.321681 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 35) took 1437 ms. Trace:
I20250626 19:57:35.325069 5353 raft_consensus.cc:357] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:35.321976 5433 rpcz_store.cc:276] 0626 19:57:33.884085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.884231 (+ 146us) service_pool.cc:225] Handling call
0626 19:57:35.023987 (+1139756us) ts_tablet_manager.cc:1145] Deleting tablet 512dc911bfce49b29c6fc8cd90523723
0626 19:57:35.024023 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:35.096883 (+ 72860us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x7b5c0003090c. stack: 00007fad7a549483 00007fad7a549380 00007fad7a0c5574 00000000004fa174 00000000004fa109 00007fad81863d11 00007fad81a04439 00007fad824ff9f4 00007fad8252fb70 00007fad82516f4d 00007fad82516ef9 00007fad82516e89 00007fad82516e51 00007fad8251607d 000000000059bcd4 000000000059bb09
0626 19:57:35.261184 (+164301us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:35.317517 (+ 56333us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:35.321634 (+ 4117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":92252544,"tablet-delete.queue_time_us":1139029}
I20250626 19:57:35.327489 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 36) took 1441 ms. Trace:
I20250626 19:57:35.327951 5433 rpcz_store.cc:276] 0626 19:57:33.886030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.886158 (+ 128us) service_pool.cc:225] Handling call
0626 19:57:35.327256 (+1441098us) ts_tablet_manager.cc:1145] Deleting tablet 3bd8dae5699c4725915814044d76a257
0626 19:57:35.327293 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:35.327441 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1428428}
I20250626 19:57:35.332881 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 38) took 1432 ms. Trace:
I20250626 19:57:35.333603 5433 rpcz_store.cc:276] 0626 19:57:33.900658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.900801 (+ 143us) service_pool.cc:225] Handling call
0626 19:57:35.331984 (+1431183us) ts_tablet_manager.cc:1145] Deleting tablet 1e453a4f8ca44f4a940928d9f4fcd5be
0626 19:57:35.332043 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:35.332198 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1430520}
I20250626 19:57:35.333885 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd6543699e254400b3516aece4486fe0" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:35.340348 4756 leader_election.cc:343] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:35.342514 5353 raft_consensus.cc:383] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:35.343917 5353 raft_consensus.cc:738] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:35.346096 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c0338c05a974e0f96865398722c5e8f" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:35.327913 5438 raft_consensus.cc:2241] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:35.351570 5353 consensus_queue.cc:260] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:35.346725 4816 raft_consensus.cc:2466] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:35.360872 5353 ts_tablet_manager.cc:1428] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
I20250626 19:57:35.361183 5366 tablet_bootstrap.cc:654] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:35.362864 5438 raft_consensus.cc:2270] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:35.364822 5438 ts_tablet_manager.cc:1905] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:35.371006 5431 ts_tablet_manager.cc:1918] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:35.373823 5431 log.cc:1199] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/4d8ffa9b658c413fa06d02ee6635dd1b
I20250626 19:57:35.371553 5353 tablet_bootstrap.cc:492] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:35.374646 5431 ts_tablet_manager.cc:1939] T 4d8ffa9b658c413fa06d02ee6635dd1b P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:35.378036 5429 leader_election.cc:290] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
W20250626 19:57:35.379565 5509 log.cc:927] Time spent T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20250626 19:57:35.382524 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 29) took 1605 ms. Trace:
I20250626 19:57:35.382959 5431 rpcz_store.cc:276] 0626 19:57:33.820660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.821436 (+ 776us) service_pool.cc:225] Handling call
0626 19:57:35.173726 (+1352290us) ts_tablet_manager.cc:1145] Deleting tablet 4d8ffa9b658c413fa06d02ee6635dd1b
0626 19:57:35.173804 (+ 78us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:35.333574 (+159770us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:35.370687 (+ 37113us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:35.382462 (+ 11775us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290012}
I20250626 19:57:35.387465 5506 leader_election.cc:290] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:35.391512 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 31) took 1543 ms. Trace:
I20250626 19:57:35.392341 5431 rpcz_store.cc:276] 0626 19:57:33.848396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.848495 (+ 99us) service_pool.cc:225] Handling call
0626 19:57:35.391005 (+1542510us) ts_tablet_manager.cc:1145] Deleting tablet fa93fa6c652f4aca848e1e4910fb8175
0626 19:57:35.391085 (+ 80us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:35.391424 (+ 339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1503049}
I20250626 19:57:35.391388 5523 raft_consensus.cc:491] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:35.394635 5523 raft_consensus.cc:513] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.395814 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 32) took 1546 ms. Trace:
I20250626 19:57:35.396468 5523 leader_election.cc:290] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:35.396507 5431 rpcz_store.cc:276] 0626 19:57:33.849374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.849524 (+ 150us) service_pool.cc:225] Handling call
0626 19:57:35.395075 (+1545551us) ts_tablet_manager.cc:1145] Deleting tablet df3da12971654d9abc2ee8840c4f1592
0626 19:57:35.395152 (+ 77us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:35.395726 (+ 574us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1506886}
I20250626 19:57:35.390493 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 4d8ffa9b658c413fa06d02ee6635dd1b (table default.Table_681222687426414bb0838e947d315979 [id=4419bc765f3a4ff1a104bb656bf642ea]) successfully deleted
W20250626 19:57:35.401501 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet fa93fa6c652f4aca848e1e4910fb8175 because the tablet was not found. No further retry: Not found: Tablet not found: fa93fa6c652f4aca848e1e4910fb8175
W20250626 19:57:35.402127 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet df3da12971654d9abc2ee8840c4f1592 because the tablet was not found. No further retry: Not found: Tablet not found: df3da12971654d9abc2ee8840c4f1592
I20250626 19:57:35.402987 5431 tablet_replica.cc:331] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:35.404549 5506 raft_consensus.cc:491] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:35.405205 5506 raft_consensus.cc:513] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.407142 5506 leader_election.cc:290] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:35.407450 5431 raft_consensus.cc:2241] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:35.408497 5431 raft_consensus.cc:2270] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:35.409916 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c0338c05a974e0f96865398722c5e8f" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:35.410828 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c92deaa237064dee8f866e3a7a355ca4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:35.411231 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7facb77eb579448f9533d285cba8650e" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:35.411722 4756 leader_election.cc:343] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:35.412046 5353 tablet_bootstrap.cc:654] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:35.412604 4756 leader_election.cc:343] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:35.413180 4756 leader_election.cc:343] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:35.414348 5431 ts_tablet_manager.cc:1905] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:35.443145 5506 raft_consensus.cc:491] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:35.444578 5506 raft_consensus.cc:513] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.436977 5353 tablet_bootstrap.cc:492] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:35.449883 5353 ts_tablet_manager.cc:1397] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.078s user 0.006s sys 0.005s
I20250626 19:57:35.463176 4956 tablet_service.cc:1468] Processing CreateTablet for tablet e7c352224d0a46c6a922bf0a0d765f17 (DEFAULT_TABLE table=default.Table_822bad36b81c42ac9101bcab4aaca9a4 [id=90440da2b42248e98586aa690f9199bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.464006 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 44960f8db08942879cde6f96e9c620da (table default.Table_eff15db1de2e4aa3977a66f4af2b3dcd [id=8df233b9523c49adb3d24727bda24b9e]). Expected version 2 got 0
I20250626 19:57:35.464205 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7c352224d0a46c6a922bf0a0d765f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.465039 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet a3b05549a1434a1fa3d337fdfb96a131 (table default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]). Expected version 1 got 0
W20250626 19:57:35.339262 4611 leader_election.cc:343] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:35.467466 5353 raft_consensus.cc:357] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:35.469200 4609 leader_election.cc:304] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:35.469774 5353 raft_consensus.cc:383] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250626 19:57:35.469815 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 107495us
I20250626 19:57:35.470661 5448 raft_consensus.cc:2802] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:35.471001 5448 raft_consensus.cc:491] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:35.471097 5353 raft_consensus.cc:738] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:35.471283 5448 raft_consensus.cc:3058] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:35.472209 5353 consensus_queue.cc:260] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:35.479869 5448 raft_consensus.cc:513] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.482661 5448 leader_election.cc:290] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:35.479002 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:35.486233 4958 tablet_service.cc:1468] Processing CreateTablet for tablet 36997dab0ab3430bba63a90805759b30 (DEFAULT_TABLE table=default.Table_c010e84626544214a45a911f02c4d903 [id=1b4629c56e5c4bf3b69fabbfb10b8008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.305208 5410 consensus_queue.cc:260] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.487535 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36997dab0ab3430bba63a90805759b30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.488665 5410 ts_tablet_manager.cc:1428] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.187s user 0.004s sys 0.000s
I20250626 19:57:35.489223 5353 ts_tablet_manager.cc:1428] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.039s user 0.000s sys 0.005s
I20250626 19:57:35.489723 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5e20ea7c17c34fbd9c117275e3c14466 (table default.Table_84725c81a9a9432c8232f87fb7627f6a [id=089059dab32a41179048a6eb4473a99a]). Expected version 1 got 0
W20250626 19:57:35.489357 4611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 150188us
I20250626 19:57:35.490073 5353 tablet_bootstrap.cc:492] T 30f1d01851be485f9d766784855ba627 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:35.493180 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 211f6137d9b84afa9aee6432440f7eb8 (DEFAULT_TABLE table=default.Table_34846ed4a33e42aa8811e45b8d029a8d [id=bc1d2774bee5476db680f3ef8db3b939]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.494091 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 211f6137d9b84afa9aee6432440f7eb8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:35.463805 4756 consensus_peers.cc:487] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:35.482983 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c0338c05a974e0f96865398722c5e8f" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:35.497845 4816 raft_consensus.cc:3058] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:35.501235 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 211f6137d9b84afa9aee6432440f7eb8 (DEFAULT_TABLE table=default.Table_34846ed4a33e42aa8811e45b8d029a8d [id=bc1d2774bee5476db680f3ef8db3b939]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.508152 5353 tablet_bootstrap.cc:654] T 30f1d01851be485f9d766784855ba627 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:35.511046 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 211f6137d9b84afa9aee6432440f7eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.513010 4816 raft_consensus.cc:2466] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:35.524364 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 512dc911bfce49b29c6fc8cd90523723 (table default.Table_89c4b3541e7c4b08b25f28abf471d42d [id=f0d69891ab4745dbb68ea3c529d1a107]) successfully deleted
W20250626 19:57:35.525449 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 3bd8dae5699c4725915814044d76a257 because the tablet was not found. No further retry: Not found: Tablet not found: 3bd8dae5699c4725915814044d76a257
W20250626 19:57:35.526425 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 1e453a4f8ca44f4a940928d9f4fcd5be because the tablet was not found. No further retry: Not found: Tablet not found: 1e453a4f8ca44f4a940928d9f4fcd5be
I20250626 19:57:35.488648 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 36997dab0ab3430bba63a90805759b30 (DEFAULT_TABLE table=default.Table_c010e84626544214a45a911f02c4d903 [id=1b4629c56e5c4bf3b69fabbfb10b8008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.484421 4801 tablet_service.cc:1468] Processing CreateTablet for tablet e7c352224d0a46c6a922bf0a0d765f17 (DEFAULT_TABLE table=default.Table_822bad36b81c42ac9101bcab4aaca9a4 [id=90440da2b42248e98586aa690f9199bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.527851 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36997dab0ab3430bba63a90805759b30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.528532 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7c352224d0a46c6a922bf0a0d765f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.529451 4609 leader_election.cc:304] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:35.537267 5524 raft_consensus.cc:2802] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:35.539124 5524 raft_consensus.cc:695] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:35.543831 5524 consensus_queue.cc:237] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:35.548352 4756 consensus_peers.cc:487] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:35.509783 4950 tablet_service.cc:1468] Processing CreateTablet for tablet e0bcbb044d1549c88e5b83eeb5c709af (DEFAULT_TABLE table=default.Table_8ed57e01e9b54a829b11bc477378d778 [id=8e37b6f1897b487db295c106a6a8aa3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.549692 4950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0bcbb044d1549c88e5b83eeb5c709af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.557166 4956 tablet_service.cc:1468] Processing CreateTablet for tablet e8122b26848141a39d2462dd3491f9cd (DEFAULT_TABLE table=default.Table_ef61f7b59da5409ca6ba9ab8d9ecf944 [id=c415b2abfcac474985b675f0ac34d64c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.557813 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7391b9eaea3e426ba4118fc8b53b72a6" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:35.558367 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8122b26848141a39d2462dd3491f9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.559235 5506 leader_election.cc:290] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
W20250626 19:57:35.559326 4756 leader_election.cc:343] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:35.565389 4798 tablet_service.cc:1468] Processing CreateTablet for tablet e8122b26848141a39d2462dd3491f9cd (DEFAULT_TABLE table=default.Table_ef61f7b59da5409ca6ba9ab8d9ecf944 [id=c415b2abfcac474985b675f0ac34d64c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.566372 4798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8122b26848141a39d2462dd3491f9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.566316 4957 tablet_service.cc:1468] Processing CreateTablet for tablet 6cc7a336385f499ca19566d5f2901e98 (DEFAULT_TABLE table=default.Table_a55a71930b3e4026aaa4b8515291c5f9 [id=84ff43fd58d54feb867bf7ddb63b4ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.567276 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cc7a336385f499ca19566d5f2901e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.568169 4394 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet a3b05549a1434a1fa3d337fdfb96a131 (table default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]). Expected version 1 got 0
I20250626 19:57:35.569427 4394 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:35.572156 4394 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:35.587168 4958 tablet_service.cc:1468] Processing CreateTablet for tablet 5706841a14bd4ece965098b88ccb7ec8 (DEFAULT_TABLE table=default.Table_428898f0f296481cb11e678bc15af894 [id=e2f7b88faf5347659fcaae53c0c9e8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.588340 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5706841a14bd4ece965098b88ccb7ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.573522 4394 catalog_manager.cc:5582] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:35.589330 4394 catalog_manager.cc:5582] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:35.590436 4394 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 5e20ea7c17c34fbd9c117275e3c14466 (table default.Table_84725c81a9a9432c8232f87fb7627f6a [id=089059dab32a41179048a6eb4473a99a]). Expected version 1 got 0
I20250626 19:57:35.592000 4795 tablet_service.cc:1468] Processing CreateTablet for tablet e0bcbb044d1549c88e5b83eeb5c709af (DEFAULT_TABLE table=default.Table_8ed57e01e9b54a829b11bc477378d778 [id=8e37b6f1897b487db295c106a6a8aa3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.593047 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0bcbb044d1549c88e5b83eeb5c709af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.595825 4796 tablet_service.cc:1468] Processing CreateTablet for tablet 6cc7a336385f499ca19566d5f2901e98 (DEFAULT_TABLE table=default.Table_a55a71930b3e4026aaa4b8515291c5f9 [id=84ff43fd58d54feb867bf7ddb63b4ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.596889 4796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cc7a336385f499ca19566d5f2901e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.597002 5353 tablet_bootstrap.cc:492] T 30f1d01851be485f9d766784855ba627 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:35.597400 5353 ts_tablet_manager.cc:1397] T 30f1d01851be485f9d766784855ba627 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.107s user 0.009s sys 0.000s
I20250626 19:57:35.599910 5353 raft_consensus.cc:357] T 30f1d01851be485f9d766784855ba627 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:35.600483 5353 raft_consensus.cc:383] T 30f1d01851be485f9d766784855ba627 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:35.600725 5353 raft_consensus.cc:738] T 30f1d01851be485f9d766784855ba627 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:35.607409 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 5706841a14bd4ece965098b88ccb7ec8 (DEFAULT_TABLE table=default.Table_428898f0f296481cb11e678bc15af894 [id=e2f7b88faf5347659fcaae53c0c9e8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.608441 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5706841a14bd4ece965098b88ccb7ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.611089 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 5c113e03594b428a979435618fe373c6 (DEFAULT_TABLE table=default.Table_ad656f771f554bc98b2dc4db559e3acb [id=3c86bc87a4a64fd8af0af50c516a2242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.611411 5431 ts_tablet_manager.cc:1918] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:35.611817 5431 log.cc:1199] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/ea7374ad6ba64e46b7df9852fa33f145
I20250626 19:57:35.612305 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c113e03594b428a979435618fe373c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.612681 5431 ts_tablet_manager.cc:1939] T ea7374ad6ba64e46b7df9852fa33f145 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:35.569317 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c0338c05a974e0f96865398722c5e8f" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:35.570248 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd6543699e254400b3516aece4486fe0" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:35.574324 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7391b9eaea3e426ba4118fc8b53b72a6" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:35.618320 4668 raft_consensus.cc:2466] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:35.578033 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7facb77eb579448f9533d285cba8650e" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:35.619491 4671 raft_consensus.cc:2466] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:35.580335 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 37) took 1680 ms. Trace:
I20250626 19:57:35.620240 5433 rpcz_store.cc:276] 0626 19:57:33.899960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.901582 (+ 1622us) service_pool.cc:225] Handling call
0626 19:57:35.344264 (+1442682us) ts_tablet_manager.cc:1145] Deleting tablet 4d8ffa9b658c413fa06d02ee6635dd1b
0626 19:57:35.571991 (+227727us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b5800000098. stack: 00007fad7a549483 00007fad7a549380 00007fad7a0c5574 00000000004fa174 00000000004fa109 00007fad7a3be9a1 00007fad7a53deb1 00007fad7a53e4e2 00007fad831da7e9 00007fad8243a968 00007fad824ff3a9 00007fad824ff9a3 00007fad8252fb70 00007fad82516f4d 00007fad82516ef9 00007fad82516e89
0626 19:57:35.580128 (+ 8137us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:35.580285 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":520754560,"tablet-delete.queue_time_us":1442143}
I20250626 19:57:35.621377 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 39) took 1708 ms. Trace:
I20250626 19:57:35.621671 5433 rpcz_store.cc:276] 0626 19:57:33.913450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.915833 (+ 2383us) service_pool.cc:225] Handling call
0626 19:57:35.621146 (+1705313us) ts_tablet_manager.cc:1145] Deleting tablet 04b0f44c1b564fdf98408881678a90a1
0626 19:57:35.621191 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:35.621331 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1704739}
I20250626 19:57:35.622565 5433 tablet_replica.cc:331] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:35.580689 4672 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c92deaa237064dee8f866e3a7a355ca4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:35.622992 4672 raft_consensus.cc:2466] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:35.623055 4394 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 3c03691d021e4ee19d5bb287ab633a3c (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:35.624254 4954 tablet_service.cc:1468] Processing CreateTablet for tablet 1ba78c1b40ef4925b4108e1455bcfc44 (DEFAULT_TABLE table=default.Table_f93f48d328f04df8b27e717e9481fd8c [id=a7f0b2ac8bd5459b93882f1bf59a9ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.629813 4942 tablet_service.cc:1468] Processing CreateTablet for tablet 1474bfa74a6346bf9e3ff49019f6c9b3 (DEFAULT_TABLE table=default.Table_fde5b6bd4ee8417e85ffccad3bd2a581 [id=ab0bf254569748ba8d31931d4bfb105d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.629994 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 33) took 1779 ms. Trace:
I20250626 19:57:35.631243 5431 rpcz_store.cc:276] 0626 19:57:33.850379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.850525 (+ 146us) service_pool.cc:225] Handling call
0626 19:57:35.400570 (+1550045us) ts_tablet_manager.cc:1145] Deleting tablet ea7374ad6ba64e46b7df9852fa33f145
0626 19:57:35.400754 (+ 184us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:35.495850 (+ 95096us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:35.581438 (+ 85588us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:35.629931 (+ 48493us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1511944}
I20250626 19:57:35.632299 5433 raft_consensus.cc:2241] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:35.631909 5353 consensus_queue.cc:260] T 30f1d01851be485f9d766784855ba627 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:35.632490 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet ea7374ad6ba64e46b7df9852fa33f145 (table default.Table_3f8afab3426b4088b717c05b65bb84e5 [id=ca71ef30f2ff4908b3b558d31e1cf989]) successfully deleted
I20250626 19:57:35.632964 5433 raft_consensus.cc:2270] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:35.634580 5433 ts_tablet_manager.cc:1905] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:35.587349 5448 raft_consensus.cc:491] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:35.643154 5448 raft_consensus.cc:513] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.643748 5544 tablet.cc:1722] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:35.644272 4676 raft_consensus.cc:2466] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:35.645687 5545 tablet.cc:1722] T a3b05549a1434a1fa3d337fdfb96a131 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:35.647815 5448 leader_election.cc:290] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:35.664806 5410 tablet_bootstrap.cc:492] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:35.656666 5353 ts_tablet_manager.cc:1428] T 30f1d01851be485f9d766784855ba627 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.059s user 0.007s sys 0.000s
I20250626 19:57:35.667501 4794 tablet_service.cc:1468] Processing CreateTablet for tablet 5c113e03594b428a979435618fe373c6 (DEFAULT_TABLE table=default.Table_ad656f771f554bc98b2dc4db559e3acb [id=3c86bc87a4a64fd8af0af50c516a2242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.657624 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c0338c05a974e0f96865398722c5e8f" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:35.679802 4654 tablet_service.cc:1468] Processing CreateTablet for tablet e7c352224d0a46c6a922bf0a0d765f17 (DEFAULT_TABLE table=default.Table_822bad36b81c42ac9101bcab4aaca9a4 [id=90440da2b42248e98586aa690f9199bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.680835 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7c352224d0a46c6a922bf0a0d765f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.687928 4651 tablet_service.cc:1468] Processing CreateTablet for tablet 36997dab0ab3430bba63a90805759b30 (DEFAULT_TABLE table=default.Table_c010e84626544214a45a911f02c4d903 [id=1b4629c56e5c4bf3b69fabbfb10b8008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.688141 4652 tablet_service.cc:1468] Processing CreateTablet for tablet 211f6137d9b84afa9aee6432440f7eb8 (DEFAULT_TABLE table=default.Table_34846ed4a33e42aa8811e45b8d029a8d [id=bc1d2774bee5476db680f3ef8db3b939]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.689069 4651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36997dab0ab3430bba63a90805759b30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.689196 4656 tablet_service.cc:1468] Processing CreateTablet for tablet e0bcbb044d1549c88e5b83eeb5c709af (DEFAULT_TABLE table=default.Table_8ed57e01e9b54a829b11bc477378d778 [id=8e37b6f1897b487db295c106a6a8aa3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.690393 4653 tablet_service.cc:1468] Processing CreateTablet for tablet e8122b26848141a39d2462dd3491f9cd (DEFAULT_TABLE table=default.Table_ef61f7b59da5409ca6ba9ab8d9ecf944 [id=c415b2abfcac474985b675f0ac34d64c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.691496 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0bcbb044d1549c88e5b83eeb5c709af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.693106 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 211f6137d9b84afa9aee6432440f7eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.699000 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8122b26848141a39d2462dd3491f9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.701617 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba78c1b40ef4925b4108e1455bcfc44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.718322 4942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1474bfa74a6346bf9e3ff49019f6c9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.723665 4648 tablet_service.cc:1468] Processing CreateTablet for tablet 6cc7a336385f499ca19566d5f2901e98 (DEFAULT_TABLE table=default.Table_a55a71930b3e4026aaa4b8515291c5f9 [id=84ff43fd58d54feb867bf7ddb63b4ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.724856 4648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cc7a336385f499ca19566d5f2901e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.726900 4793 tablet_service.cc:1468] Processing CreateTablet for tablet 1474bfa74a6346bf9e3ff49019f6c9b3 (DEFAULT_TABLE table=default.Table_fde5b6bd4ee8417e85ffccad3bd2a581 [id=ab0bf254569748ba8d31931d4bfb105d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.728111 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd6543699e254400b3516aece4486fe0" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:35.728724 4816 raft_consensus.cc:2466] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:35.728813 4755 leader_election.cc:304] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:35.668551 4797 tablet_service.cc:1468] Processing CreateTablet for tablet 1ba78c1b40ef4925b4108e1455bcfc44 (DEFAULT_TABLE table=default.Table_f93f48d328f04df8b27e717e9481fd8c [id=a7f0b2ac8bd5459b93882f1bf59a9ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.730579 4609 leader_election.cc:304] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:35.730832 4794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c113e03594b428a979435618fe373c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.732446 5353 tablet_bootstrap.cc:492] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:35.735064 5431 tablet_replica.cc:331] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:35.739100 5431 raft_consensus.cc:2241] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:35.739739 5431 raft_consensus.cc:2270] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:35.741900 5431 ts_tablet_manager.cc:1905] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:35.742934 5524 raft_consensus.cc:2802] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:35.743342 5524 raft_consensus.cc:491] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:35.743736 5524 raft_consensus.cc:3058] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:35.738270 5353 tablet_bootstrap.cc:654] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:35.749341 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 5706841a14bd4ece965098b88ccb7ec8 (DEFAULT_TABLE table=default.Table_428898f0f296481cb11e678bc15af894 [id=e2f7b88faf5347659fcaae53c0c9e8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.750370 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5706841a14bd4ece965098b88ccb7ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.751343 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet a3b05549a1434a1fa3d337fdfb96a131 (table default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]). Expected version 1 got 0
I20250626 19:57:35.751314 4649 tablet_service.cc:1468] Processing CreateTablet for tablet 5c113e03594b428a979435618fe373c6 (DEFAULT_TABLE table=default.Table_ad656f771f554bc98b2dc4db559e3acb [id=3c86bc87a4a64fd8af0af50c516a2242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.752450 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c113e03594b428a979435618fe373c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.760424 4657 tablet_service.cc:1468] Processing CreateTablet for tablet 1ba78c1b40ef4925b4108e1455bcfc44 (DEFAULT_TABLE table=default.Table_f93f48d328f04df8b27e717e9481fd8c [id=a7f0b2ac8bd5459b93882f1bf59a9ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.761499 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba78c1b40ef4925b4108e1455bcfc44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.772320 5353 tablet_bootstrap.cc:492] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:35.772727 5353 ts_tablet_manager.cc:1397] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.002s
I20250626 19:57:35.774960 5353 raft_consensus.cc:357] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.775542 5353 raft_consensus.cc:383] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:35.775796 5353 raft_consensus.cc:738] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:35.776419 5353 consensus_queue.cc:260] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.751969 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:35.779182 5353 ts_tablet_manager.cc:1428] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20250626 19:57:35.784641 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5e20ea7c17c34fbd9c117275e3c14466 (table default.Table_84725c81a9a9432c8232f87fb7627f6a [id=089059dab32a41179048a6eb4473a99a]). Expected version 1 got 0
I20250626 19:57:35.791538 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:35.794598 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 3c03691d021e4ee19d5bb287ab633a3c (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
W20250626 19:57:35.801852 5535 log.cc:927] Time spent T a3b05549a1434a1fa3d337fdfb96a131 P 93f885c6802341c889a40e9c20fbfff1: Append to log took a long time: real 0.177s user 0.001s sys 0.000s
I20250626 19:57:35.801831 5448 consensus_queue.cc:1035] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:35.802856 5353 tablet_bootstrap.cc:492] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:35.810830 4793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1474bfa74a6346bf9e3ff49019f6c9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.813992 5416 raft_consensus.cc:2747] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:35.814936 4797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba78c1b40ef4925b4108e1455bcfc44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.821542 4973 raft_consensus.cc:3058] T 30f1d01851be485f9d766784855ba627 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:35.837941 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 4d8ffa9b658c413fa06d02ee6635dd1b because the tablet was not found. No further retry: Not found: Tablet not found: 4d8ffa9b658c413fa06d02ee6635dd1b
W20250626 19:57:35.833786 4755 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 105109us
I20250626 19:57:35.840484 4653 tablet_service.cc:1468] Processing CreateTablet for tablet 1474bfa74a6346bf9e3ff49019f6c9b3 (DEFAULT_TABLE table=default.Table_fde5b6bd4ee8417e85ffccad3bd2a581 [id=ab0bf254569748ba8d31931d4bfb105d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:35.841966 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1474bfa74a6346bf9e3ff49019f6c9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:35.845224 5353 tablet_bootstrap.cc:654] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:35.848672 5438 ts_tablet_manager.cc:1918] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:35.849078 5438 log.cc:1199] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/78a8f494f8614e7ebb8ca880b6c80e78
I20250626 19:57:35.849781 5438 ts_tablet_manager.cc:1939] T 78a8f494f8614e7ebb8ca880b6c80e78 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
W20250626 19:57:35.851202 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 04b0f44c1b564fdf98408881678a90a1 because the tablet was not found. No further retry: Not found: Tablet not found: 04b0f44c1b564fdf98408881678a90a1
I20250626 19:57:35.856673 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd6543699e254400b3516aece4486fe0" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:35.860888 5431 ts_tablet_manager.cc:1918] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:57:35.870939 5431 log.cc:1199] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/5785508e37624f02bc2f6cc93007e3f8
I20250626 19:57:35.871927 5431 ts_tablet_manager.cc:1939] T 5785508e37624f02bc2f6cc93007e3f8 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:35.839533 4755 leader_election.cc:304] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:35.876839 5353 tablet_bootstrap.cc:492] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:35.877282 5353 ts_tablet_manager.cc:1397] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.000s
I20250626 19:57:35.885936 5407 raft_consensus.cc:491] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:35.886325 5407 raft_consensus.cc:513] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:35.888113 5407 leader_election.cc:290] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:35.889624 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c990e4eca5cf4a8e8295e9599cf5be29" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:35.890344 4671 raft_consensus.cc:2391] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:35.894984 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 34) took 2043 ms. Trace:
I20250626 19:57:35.896440 5431 rpcz_store.cc:276] 0626 19:57:33.851926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.852111 (+ 185us) service_pool.cc:225] Handling call
0626 19:57:35.640043 (+1787932us) ts_tablet_manager.cc:1145] Deleting tablet 5785508e37624f02bc2f6cc93007e3f8
0626 19:57:35.731685 (+ 91642us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:35.787268 (+ 55583us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:35.860656 (+ 73388us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:35.894921 (+ 34265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1751424}
I20250626 19:57:35.900283 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 5785508e37624f02bc2f6cc93007e3f8 (table default.Table_b401c8a5ab6744029f693b1b5e0f306d [id=95832380520e4cf18d762654fb684db3]) successfully deleted
I20250626 19:57:35.901244 5431 tablet_replica.cc:331] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:35.902087 5431 raft_consensus.cc:2241] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:35.905633 5433 ts_tablet_manager.cc:1918] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:35.905977 5433 log.cc:1199] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/499f6ae3262940b4859c14a53722482f
I20250626 19:57:35.905396 5407 raft_consensus.cc:491] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:35.906692 5433 ts_tablet_manager.cc:1939] T 499f6ae3262940b4859c14a53722482f P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:35.906972 5407 raft_consensus.cc:513] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.908485 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 40) took 1855 ms. Trace:
I20250626 19:57:35.908751 5433 rpcz_store.cc:276] 0626 19:57:34.052655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:34.052794 (+ 139us) service_pool.cc:225] Handling call
0626 19:57:35.622417 (+1569623us) ts_tablet_manager.cc:1145] Deleting tablet 499f6ae3262940b4859c14a53722482f
0626 19:57:35.622451 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:35.641945 (+ 19494us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:35.905428 (+263483us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:35.908434 (+ 3006us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5770,"tablet-delete.queue_time_us":1569069}
I20250626 19:57:35.909317 5407 leader_election.cc:290] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:35.910012 5431 raft_consensus.cc:2270] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:35.912808 5431 ts_tablet_manager.cc:1905] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:35.910513 5353 raft_consensus.cc:357] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.913224 5353 raft_consensus.cc:383] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:35.913519 5353 raft_consensus.cc:738] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:35.914170 5353 consensus_queue.cc:260] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.918300 5353 ts_tablet_manager.cc:1428] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20250626 19:57:35.920900 5353 tablet_bootstrap.cc:492] T 33a75d223c274c049cf25f79fcc4b961 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:35.925923 5353 tablet_bootstrap.cc:654] T 33a75d223c274c049cf25f79fcc4b961 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:35.945225 5353 tablet_bootstrap.cc:492] T 33a75d223c274c049cf25f79fcc4b961 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:35.945712 5353 ts_tablet_manager.cc:1397] T 33a75d223c274c049cf25f79fcc4b961 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.002s
I20250626 19:57:35.949214 5353 raft_consensus.cc:357] T 33a75d223c274c049cf25f79fcc4b961 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.949970 5353 raft_consensus.cc:383] T 33a75d223c274c049cf25f79fcc4b961 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:35.950260 5353 raft_consensus.cc:738] T 33a75d223c274c049cf25f79fcc4b961 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
W20250626 19:57:35.954413 4756 consensus_peers.cc:487] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5785508e37624f02bc2f6cc93007e3f8. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:35.955516 5353 consensus_queue.cc:260] T 33a75d223c274c049cf25f79fcc4b961 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.884030 5506 raft_consensus.cc:2802] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:35.964121 5353 ts_tablet_manager.cc:1428] T 33a75d223c274c049cf25f79fcc4b961 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250626 19:57:35.967193 5353 tablet_bootstrap.cc:492] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:35.967437 5550 tablet.cc:1722] T a3b05549a1434a1fa3d337fdfb96a131 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250626 19:57:35.941548 4755 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 102247us
I20250626 19:57:35.974507 5431 ts_tablet_manager.cc:1918] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:35.974889 5431 log.cc:1199] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/afd1fafb53434957b002fad5c77431d4
I20250626 19:57:35.974656 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c92deaa237064dee8f866e3a7a355ca4" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:35.976528 4815 raft_consensus.cc:2466] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
W20250626 19:57:35.979619 4911 consensus_peers.cc:487] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Couldn't send request to peer 93f885c6802341c889a40e9c20fbfff1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78a8f494f8614e7ebb8ca880b6c80e78. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:35.987426 4911 leader_election.cc:304] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:35.988122 5407 raft_consensus.cc:2802] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:35.988442 5407 raft_consensus.cc:491] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:35.988725 5407 raft_consensus.cc:3058] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:35.991272 4755 leader_election.cc:304] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:35.992393 4755 leader_election.cc:304] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:35.993518 5429 raft_consensus.cc:2802] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:35.993849 5429 raft_consensus.cc:491] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:35.994141 5429 raft_consensus.cc:3058] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:35.994882 5523 raft_consensus.cc:2802] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:35.995200 5523 raft_consensus.cc:491] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:35.995543 5523 raft_consensus.cc:3058] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:35.963485 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c990e4eca5cf4a8e8295e9599cf5be29" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:35.996915 4820 raft_consensus.cc:2391] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 93f885c6802341c889a40e9c20fbfff1 in this term.
I20250626 19:57:36.002321 5407 raft_consensus.cc:513] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.003629 5431 ts_tablet_manager.cc:1939] T afd1fafb53434957b002fad5c77431d4 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:35.962205 5506 raft_consensus.cc:491] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:36.007195 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 35) took 2153 ms. Trace:
I20250626 19:57:36.007510 5431 rpcz_store.cc:276] 0626 19:57:33.853320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.853763 (+ 443us) service_pool.cc:225] Handling call
0626 19:57:35.901082 (+2047319us) ts_tablet_manager.cc:1145] Deleting tablet afd1fafb53434957b002fad5c77431d4
0626 19:57:35.901129 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:35.937609 (+ 36480us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:35.960976 (+ 23367us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.007132 (+ 46156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2012241}
I20250626 19:57:36.008582 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 37) took 2151 ms. Trace:
I20250626 19:57:36.008894 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet afd1fafb53434957b002fad5c77431d4 (table default.Table_b6e3c1f4224b4dee8c044d2548276942 [id=cb6e1a290c9c47c0bef9de879ab06bd9]) successfully deleted
I20250626 19:57:36.008855 5431 rpcz_store.cc:276] 0626 19:57:33.857614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.858874 (+ 1260us) service_pool.cc:225] Handling call
0626 19:57:36.008344 (+2149470us) ts_tablet_manager.cc:1145] Deleting tablet 3bd8dae5699c4725915814044d76a257
0626 19:57:36.008383 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:36.008538 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2119156}
I20250626 19:57:36.009706 5431 tablet_replica.cc:331] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
W20250626 19:57:36.010402 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 3bd8dae5699c4725915814044d76a257 because the tablet was not found. No further retry: Not found: Tablet not found: 3bd8dae5699c4725915814044d76a257
I20250626 19:57:36.010459 5431 raft_consensus.cc:2241] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:36.006851 5506 raft_consensus.cc:3058] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:36.014912 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c92deaa237064dee8f866e3a7a355ca4" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:36.016062 5407 leader_election.cc:290] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:35.986847 5353 tablet_bootstrap.cc:654] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:36.017035 4391 catalog_manager.cc:5582] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:36.018672 5431 raft_consensus.cc:2270] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:36.019301 4942 tablet_service.cc:1468] Processing CreateTablet for tablet e65b16141c3d47c5b66024da59b9bbe9 (DEFAULT_TABLE table=default.Table_bca98e7f86544ed89dbd4af5356026ca [id=d1d1bf84503e46afa724f4aa0bb60584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.020380 4942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65b16141c3d47c5b66024da59b9bbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.020387 5431 ts_tablet_manager.cc:1905] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:36.019976 4391 catalog_manager.cc:5582] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:36.023820 4391 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet a3b05549a1434a1fa3d337fdfb96a131 (table default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b]). Expected version 1 got 0
I20250626 19:57:36.024933 4391 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:36.025790 4391 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:36.026520 4391 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 5e20ea7c17c34fbd9c117275e3c14466 (table default.Table_84725c81a9a9432c8232f87fb7627f6a [id=089059dab32a41179048a6eb4473a99a]). Expected version 1 got 0
I20250626 19:57:36.027498 4391 catalog_manager.cc:5582] T 30f1d01851be485f9d766784855ba627 P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:36.029132 4954 tablet_service.cc:1468] Processing CreateTablet for tablet a3636b78f4a64f9eacb4e5979768929b (DEFAULT_TABLE table=default.Table_07d4c194b22c4529b42fc16c2ebaff31 [id=c934f92aa6c9402ba2d0bae8a0288958]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.029500 4391 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 3c03691d021e4ee19d5bb287ab633a3c (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:36.031369 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3636b78f4a64f9eacb4e5979768929b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.031445 4397 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34796 (request call id 78) took 1956 ms. Trace:
I20250626 19:57:36.031795 4397 rpcz_store.cc:276] 0626 19:57:34.075369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:34.077636 (+ 2267us) service_pool.cc:225] Handling call
0626 19:57:34.107511 (+ 29875us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:34.108204 (+ 693us) catalog_manager.cc:3683] Apply alter schema
0626 19:57:34.108750 (+ 546us) catalog_manager.cc:3715] Acquired catalog manager lock
0626 19:57:34.108918 (+ 168us) catalog_manager.cc:3873] Updating metadata on disk
0626 19:57:34.163516 (+ 54598us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:34.163544 (+ 28us) write_op.cc:454] Released schema lock
0626 19:57:34.163754 (+ 210us) catalog_manager.cc:3904] Committing alterations to in-memory state
0626 19:57:36.031380 (+1867626us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:34.110120 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:34.110482 (+ 362us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:34.110508 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:34.110520 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:34.110779 (+ 259us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:34.110821 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:34.110847 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:34.110990 (+ 143us) tablet.cc:699] Row locks acquired
0626 19:57:34.111004 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:57:34.111103 (+ 99us) write_op.cc:270] Start()
0626 19:57:34.111181 (+ 78us) write_op.cc:276] Timestamp: P: 1750967854111185 usec, L: 0
0626 19:57:34.111198 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:34.111683 (+ 485us) log.cc:844] Serialized 405 byte log entry
0626 19:57:34.123472 (+ 11789us) op_driver.cc:464] REPLICATION: finished
0626 19:57:34.123967 (+ 495us) write_op.cc:301] APPLY: starting
0626 19:57:34.124021 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:34.124110 (+ 89us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:34.124193 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:34.124353 (+ 160us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:34.124475 (+ 122us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:34.124515 (+ 40us) write_op.cc:312] APPLY: finished
0626 19:57:34.124678 (+ 163us) log.cc:844] Serialized 23 byte log entry
0626 19:57:34.124807 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:34.124868 (+ 61us) write_op.cc:454] Released schema lock
0626 19:57:34.124896 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":342,"apply.run_cpu_time_us":1097,"apply.run_wall_time_us":1096,"num_ops":1,"prepare.queue_time_us":92,"prepare.run_cpu_time_us":1855,"prepare.run_wall_time_us":1855,"raft.queue_time_us":3024,"raft.run_cpu_time_us":611,"raft.run_wall_time_us":611,"replication_time_us":12184}]]}
I20250626 19:57:36.036283 4397 master_service.cc:948] ReplaceTablet: received request to replace tablet 2ab0320af47049cca3ca87172b9fe5e5 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:36.037158 4390 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet c92deaa237064dee8f866e3a7a355ca4
I20250626 19:57:36.038570 4390 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34796 (request call id 74) took 2352 ms. Trace:
I20250626 19:57:36.039495 4911 leader_election.cc:304] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:36.040464 5407 raft_consensus.cc:3058] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:36.038903 4390 rpcz_store.cc:276] 0626 19:57:33.686872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.861906 (+175034us) service_pool.cc:225] Handling call
0626 19:57:34.056282 (+194376us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:34.058058 (+ 1776us) catalog_manager.cc:2790] Modifying in-memory table state
0626 19:57:34.058193 (+ 135us) catalog_manager.cc:2799] Locking tablets
0626 19:57:34.255914 (+197721us) catalog_manager.cc:2813] Removing table and tablets from system table
0626 19:57:34.429320 (+173406us) catalog_manager.cc:2831] Removing table from by-name map
0626 19:57:34.429542 (+ 222us) catalog_manager.cc:2847] Committing in-memory state
0626 19:57:36.037075 (+1607533us) catalog_manager.cc:2851] Aborting table tasks
0626 19:57:36.038504 (+ 1429us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:34.258592 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:34.258993 (+ 401us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:34.259021 (+ 28us) write_op.cc:435] Acquired schema lock
0626 19:57:34.259037 (+ 16us) tablet.cc:660] Decoding operations
0626 19:57:34.259338 (+ 301us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:34.259376 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:34.259399 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:34.259641 (+ 242us) tablet.cc:699] Row locks acquired
0626 19:57:34.259655 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:57:34.259756 (+ 101us) write_op.cc:270] Start()
0626 19:57:34.259832 (+ 76us) write_op.cc:276] Timestamp: P: 1750967854259899 usec, L: 0
0626 19:57:34.259848 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:57:34.260344 (+ 496us) log.cc:844] Serialized 728 byte log entry
0626 19:57:34.374974 (+114630us) op_driver.cc:464] REPLICATION: finished
0626 19:57:34.426959 (+ 51985us) write_op.cc:301] APPLY: starting
0626 19:57:34.427016 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:34.427145 (+ 129us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:34.427783 (+ 638us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:34.428026 (+ 243us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:34.428178 (+ 152us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:34.428222 (+ 44us) write_op.cc:312] APPLY: finished
0626 19:57:34.428409 (+ 187us) log.cc:844] Serialized 29 byte log entry
0626 19:57:34.428998 (+ 589us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:34.429080 (+ 82us) write_op.cc:454] Released schema lock
0626 19:57:34.429115 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:34.429760 (+ 645us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:34.429786 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51768,"apply.run_cpu_time_us":1947,"apply.run_wall_time_us":2792,"num_ops":2,"prepare.queue_time_us":251,"prepare.run_cpu_time_us":2075,"prepare.run_wall_time_us":18997,"raft.queue_time_us":2444,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":14,"replication_time_us":114997}]]}
I20250626 19:57:36.045159 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet c92deaa237064dee8f866e3a7a355ca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:34 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:36.045317 4390 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_ad7eb0792b7444b2891bf198ef117218"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:36.047732 5407 raft_consensus.cc:2747] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:36.047420 4390 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad7eb0792b7444b2891bf198ef117218 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:57:36.050642 4912 consensus_peers.cc:487] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 499f6ae3262940b4859c14a53722482f. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:36.050660 4798 tablet_service.cc:1515] Processing DeleteTablet for tablet c92deaa237064dee8f866e3a7a355ca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:34 UTC) from {username='slave'} at 127.0.0.1:40742
W20250626 19:57:36.051578 4396 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c113e03594b428a979435618fe373c6 consensus state has no leader
I20250626 19:57:36.052033 4396 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34796 (request call id 76) took 2335 ms. Trace:
I20250626 19:57:36.056293 5255 raft_consensus.cc:491] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:36.056520 4755 leader_election.cc:304] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:36.056704 5255 raft_consensus.cc:513] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:36.058431 5255 leader_election.cc:290] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:36.052302 4396 rpcz_store.cc:276] 0626 19:57:33.716891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.930393 (+213502us) service_pool.cc:225] Handling call
0626 19:57:34.100822 (+170429us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:34.101513 (+ 691us) catalog_manager.cc:3683] Apply alter schema
0626 19:57:34.102062 (+ 549us) catalog_manager.cc:3715] Acquired catalog manager lock
0626 19:57:34.102278 (+ 216us) catalog_manager.cc:3873] Updating metadata on disk
0626 19:57:34.182813 (+ 80535us) catalog_manager.cc:3904] Committing alterations to in-memory state
0626 19:57:36.051979 (+1869166us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:34.153145 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:34.153534 (+ 389us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:34.153562 (+ 28us) write_op.cc:435] Acquired schema lock
0626 19:57:34.153577 (+ 15us) tablet.cc:660] Decoding operations
0626 19:57:34.153825 (+ 248us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:34.153864 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:34.153884 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:34.154028 (+ 144us) tablet.cc:699] Row locks acquired
0626 19:57:34.154041 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:57:34.154140 (+ 99us) write_op.cc:270] Start()
0626 19:57:34.154229 (+ 89us) write_op.cc:276] Timestamp: P: 1750967854154239 usec, L: 0
0626 19:57:34.154244 (+ 15us) op_driver.cc:348] REPLICATION: starting
0626 19:57:34.169480 (+ 15236us) log.cc:844] Serialized 405 byte log entry
0626 19:57:34.171088 (+ 1608us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72b481 00007fe78f7a2a18 00007fe78fc0f811 00007fe78fc0efe8 00007fe78fc13c81 00007fe78fc13c39 00007fe78fc13bc9 00007fe78fc13b91 00007fe78fc12dbd 000000000059bcd4 000000000059bb09
0626 19:57:34.179944 (+ 8856us) op_driver.cc:464] REPLICATION: finished
0626 19:57:34.180348 (+ 404us) write_op.cc:301] APPLY: starting
0626 19:57:34.180410 (+ 62us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:34.180504 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:34.180529 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:34.180691 (+ 162us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:34.180818 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:34.180859 (+ 41us) write_op.cc:312] APPLY: finished
0626 19:57:34.181057 (+ 198us) log.cc:844] Serialized 23 byte log entry
0626 19:57:34.181231 (+ 174us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:34.181293 (+ 62us) write_op.cc:454] Released schema lock
0626 19:57:34.181322 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:34.181533 (+ 211us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:34.181556 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":266,"apply.run_cpu_time_us":1146,"apply.run_wall_time_us":1145,"num_ops":1,"prepare.queue_time_us":5510,"prepare.run_cpu_time_us":3805,"prepare.run_wall_time_us":18175,"raft.queue_time_us":2922,"raft.run_cpu_time_us":644,"raft.run_wall_time_us":646,"replication_time_us":25604,"spinlock_wait_cycles":33824640}]]}
I20250626 19:57:36.061347 5416 raft_consensus.cc:2802] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:36.061702 5416 raft_consensus.cc:491] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:36.062032 5416 raft_consensus.cc:3058] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:36.062898 4396 master_service.cc:948] ReplaceTablet: received request to replace tablet 82823fd9ceb1485386616319fa7aeb43 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:36.069200 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 31) took 2364 ms. Trace:
I20250626 19:57:36.072196 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a31f1a7739544706bff2a500d6026086" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:36.074949 5431 ts_tablet_manager.cc:1918] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:36.075289 5431 log.cc:1199] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/612f87796cfa44ddbb19b02797031db9
I20250626 19:57:36.075879 5431 ts_tablet_manager.cc:1939] T 612f87796cfa44ddbb19b02797031db9 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:36.077564 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 39) took 2216 ms. Trace:
I20250626 19:57:36.077998 5431 rpcz_store.cc:276] 0626 19:57:33.860705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.952991 (+ 92286us) service_pool.cc:225] Handling call
0626 19:57:36.009575 (+2056584us) ts_tablet_manager.cc:1145] Deleting tablet 612f87796cfa44ddbb19b02797031db9
0626 19:57:36.009610 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:36.035666 (+ 26056us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.055927 (+ 20261us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.077509 (+ 21582us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2055764}
I20250626 19:57:36.079834 5431 tablet_replica.cc:331] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:36.080330 5353 tablet_bootstrap.cc:492] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:36.080493 5431 raft_consensus.cc:2241] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:36.080690 5353 ts_tablet_manager.cc:1397] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.116s user 0.020s sys 0.003s
I20250626 19:57:36.081540 5431 raft_consensus.cc:2270] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:36.083356 5431 ts_tablet_manager.cc:1905] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:36.083186 5353 raft_consensus.cc:357] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.083825 5353 raft_consensus.cc:383] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:36.084064 5353 raft_consensus.cc:738] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:36.084683 5353 consensus_queue.cc:260] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.086681 5353 ts_tablet_manager.cc:1428] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20250626 19:57:36.088264 5366 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.001s sys 0.013s
W20250626 19:57:36.088551 5366 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.719s user 0.001s sys 0.013s
I20250626 19:57:36.090449 5366 tablet_bootstrap.cc:492] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
W20250626 19:57:36.091012 4386 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5c113e03594b428a979435618fe373c6 consensus state has no leader
I20250626 19:57:36.093313 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 612f87796cfa44ddbb19b02797031db9 (table default.Table_c5ff61c31334492c9cc8dede96094236 [id=23c193073c364d7c921797f0940caf10]) successfully deleted
I20250626 19:57:36.094457 5353 tablet_bootstrap.cc:492] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:36.098944 4795 tablet_service.cc:1468] Processing CreateTablet for tablet a3636b78f4a64f9eacb4e5979768929b (DEFAULT_TABLE table=default.Table_07d4c194b22c4529b42fc16c2ebaff31 [id=c934f92aa6c9402ba2d0bae8a0288958]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.100030 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3636b78f4a64f9eacb4e5979768929b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.110396 5366 ts_tablet_manager.cc:1397] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 1.101s user 0.018s sys 0.035s
I20250626 19:57:36.119930 5505 consensus_queue.cc:1035] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:36.120400 4396 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 82823fd9ceb1485386616319fa7aeb43
I20250626 19:57:36.069525 5438 rpcz_store.cc:276] 0626 19:57:33.705119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.705660 (+ 541us) service_pool.cc:225] Handling call
0626 19:57:35.066952 (+1361292us) ts_tablet_manager.cc:1145] Deleting tablet 78a8f494f8614e7ebb8ca880b6c80e78
0626 19:57:35.086882 (+ 19930us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:35.386363 (+299481us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:35.848443 (+462080us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.069145 (+220702us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15942,"tablet-delete.queue_time_us":1325548}
I20250626 19:57:36.121305 5505 consensus_queue.cc:1035] T 33a75d223c274c049cf25f79fcc4b961 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:36.121791 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 78a8f494f8614e7ebb8ca880b6c80e78 (table default.Table_cc421ddfd27f48b2a8a3bf433f8720f8 [id=eec0405794114b30877bb326de9f9076]) successfully deleted
I20250626 19:57:36.122017 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 82823fd9ceb1485386616319fa7aeb43 with delete_type TABLET_DATA_DELETED (replaced by tablet f6ec952321924098a142f1438a283aab) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:36.122567 5438 tablet_replica.cc:331] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:36.123155 4394 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: default.Table_eff15db1de2e4aa3977a66f4af2b3dcd [id=8df233b9523c49adb3d24727bda24b9e] alter complete (version 2)
I20250626 19:57:36.123523 5438 raft_consensus.cc:2241] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:36.124359 5438 raft_consensus.cc:2270] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:36.126140 5438 ts_tablet_manager.cc:1905] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:36.127386 5353 tablet_bootstrap.cc:654] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:36.128685 5366 raft_consensus.cc:357] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:36.129317 5366 raft_consensus.cc:383] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:36.129598 5366 raft_consensus.cc:738] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:36.129356 5506 raft_consensus.cc:513] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.131747 5429 raft_consensus.cc:513] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.132896 4819 raft_consensus.cc:2391] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 93f885c6802341c889a40e9c20fbfff1 in this term.
I20250626 19:57:36.137387 5523 raft_consensus.cc:513] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.140375 5523 leader_election.cc:290] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:36.140460 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:36.142426 4393 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet a31f1a7739544706bff2a500d6026086
I20250626 19:57:36.143095 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7facb77eb579448f9533d285cba8650e" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
W20250626 19:57:36.144621 4756 leader_election.cc:343] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:36.106209 5416 raft_consensus.cc:513] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.149386 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd6543699e254400b3516aece4486fe0" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:36.153949 5431 ts_tablet_manager.cc:1918] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:36.154372 5431 log.cc:1199] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/78a8f494f8614e7ebb8ca880b6c80e78
W20250626 19:57:36.155015 4756 leader_election.cc:343] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:36.155134 4393 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34796 (request call id 75) took 2455 ms. Trace:
I20250626 19:57:36.158218 4798 tablet_service.cc:1515] Processing DeleteTablet for tablet afd1fafb53434957b002fad5c77431d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:36.158542 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 512dc911bfce49b29c6fc8cd90523723 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:36.159605 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet a31f1a7739544706bff2a500d6026086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:34 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:36.160919 5416 leader_election.cc:290] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:36.161528 4796 tablet_service.cc:1515] Processing DeleteTablet for tablet ea7374ad6ba64e46b7df9852fa33f145 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:36.163954 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c92deaa237064dee8f866e3a7a355ca4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:36.164666 4973 raft_consensus.cc:2391] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 93f885c6802341c889a40e9c20fbfff1 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:36.166179 5429 leader_election.cc:290] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:36.166826 5366 consensus_queue.cc:260] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:36.168058 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet a31f1a7739544706bff2a500d6026086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:34 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:36.170091 5366 ts_tablet_manager.cc:1428] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.059s user 0.005s sys 0.000s
I20250626 19:57:36.174211 4796 tablet_service.cc:1468] Processing CreateTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d (DEFAULT_TABLE table=default.Table_ff801b0f32eb4c948c82f6c4b7d42b97 [id=46243e06f82542189b78751aef158149]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.174340 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (DEFAULT_TABLE table=default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.174294 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (DEFAULT_TABLE table=default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.175453 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 465ed0fe56fb4960a58e92b9b3d594e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.176095 4800 tablet_service.cc:1468] Processing CreateTablet for tablet e65b16141c3d47c5b66024da59b9bbe9 (DEFAULT_TABLE table=default.Table_bca98e7f86544ed89dbd4af5356026ca [id=d1d1bf84503e46afa724f4aa0bb60584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.179623 4958 tablet_service.cc:1468] Processing CreateTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d (DEFAULT_TABLE table=default.Table_ff801b0f32eb4c948c82f6c4b7d42b97 [id=46243e06f82542189b78751aef158149]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.180655 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb34ffcd4da04e32a1220b3f298ec22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.184441 4796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb34ffcd4da04e32a1220b3f298ec22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.155221 5431 ts_tablet_manager.cc:1939] T 78a8f494f8614e7ebb8ca880b6c80e78 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:36.190335 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 465ed0fe56fb4960a58e92b9b3d594e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.192039 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 40) took 2237 ms. Trace:
I20250626 19:57:36.200145 4797 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e453a4f8ca44f4a940928d9f4fcd5be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:31 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:36.200814 5366 tablet_bootstrap.cc:492] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:36.155438 4393 rpcz_store.cc:276] 0626 19:57:33.699746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.876824 (+177078us) service_pool.cc:225] Handling call
0626 19:57:34.008460 (+131636us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:34.098536 (+ 90076us) catalog_manager.cc:2790] Modifying in-memory table state
0626 19:57:34.098679 (+ 143us) catalog_manager.cc:2799] Locking tablets
0626 19:57:34.249055 (+150376us) catalog_manager.cc:2813] Removing table and tablets from system table
0626 19:57:34.309896 (+ 60841us) catalog_manager.cc:2831] Removing table from by-name map
0626 19:57:34.310101 (+ 205us) catalog_manager.cc:2847] Committing in-memory state
0626 19:57:36.141669 (+1831568us) catalog_manager.cc:2851] Aborting table tasks
0626 19:57:36.155070 (+ 13401us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:34.251402 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:34.251833 (+ 431us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:34.251866 (+ 33us) write_op.cc:435] Acquired schema lock
0626 19:57:34.251885 (+ 19us) tablet.cc:660] Decoding operations
0626 19:57:34.252262 (+ 377us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:34.252314 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:34.252347 (+ 33us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:34.252677 (+ 330us) tablet.cc:699] Row locks acquired
0626 19:57:34.252697 (+ 20us) write_op.cc:260] PREPARE: finished
0626 19:57:34.252824 (+ 127us) write_op.cc:270] Start()
0626 19:57:34.252924 (+ 100us) write_op.cc:276] Timestamp: P: 1750967854252953 usec, L: 0
0626 19:57:34.252946 (+ 22us) op_driver.cc:348] REPLICATION: starting
0626 19:57:34.253566 (+ 620us) log.cc:844] Serialized 759 byte log entry
0626 19:57:34.305860 (+ 52294us) op_driver.cc:464] REPLICATION: finished
0626 19:57:34.307951 (+ 2091us) write_op.cc:301] APPLY: starting
0626 19:57:34.307993 (+ 42us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:34.308083 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:34.308184 (+ 101us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:34.308384 (+ 200us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:34.308509 (+ 125us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:34.308555 (+ 46us) write_op.cc:312] APPLY: finished
0626 19:57:34.308703 (+ 148us) log.cc:844] Serialized 29 byte log entry
0626 19:57:34.308822 (+ 119us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:34.308873 (+ 51us) write_op.cc:454] Released schema lock
0626 19:57:34.308905 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1970,"apply.run_cpu_time_us":1117,"apply.run_wall_time_us":1117,"num_ops":2,"prepare.queue_time_us":117,"prepare.run_cpu_time_us":2431,"prepare.run_wall_time_us":3665,"raft.queue_time_us":158,"raft.run_cpu_time_us":21,"raft.run_wall_time_us":21,"replication_time_us":52836,"thread_start_us":68479,"threads_started":1}]]}
I20250626 19:57:36.192508 5431 rpcz_store.cc:276] 0626 19:57:33.954267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.954369 (+ 102us) service_pool.cc:225] Handling call
0626 19:57:36.079024 (+2124655us) ts_tablet_manager.cc:1145] Deleting tablet 78a8f494f8614e7ebb8ca880b6c80e78
0626 19:57:36.079065 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:36.114209 (+ 35144us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.153741 (+ 39532us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.191943 (+ 38202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2123688}
I20250626 19:57:36.207137 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 41) took 2251 ms. Trace:
I20250626 19:57:36.207544 5431 rpcz_store.cc:276] 0626 19:57:33.955742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.955833 (+ 91us) service_pool.cc:225] Handling call
0626 19:57:36.206867 (+2251034us) ts_tablet_manager.cc:1145] Deleting tablet 512dc911bfce49b29c6fc8cd90523723
0626 19:57:36.206906 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:36.207083 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2250543}
I20250626 19:57:36.209971 4799 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d8ffa9b658c413fa06d02ee6635dd1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:31 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:36.210093 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 78a8f494f8614e7ebb8ca880b6c80e78 (table default.Table_cc421ddfd27f48b2a8a3bf433f8720f8 [id=eec0405794114b30877bb326de9f9076]) successfully deleted
I20250626 19:57:36.211246 5366 tablet_bootstrap.cc:654] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:36.208508 5353 tablet_bootstrap.cc:492] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:36.214138 5353 ts_tablet_manager.cc:1397] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.127s user 0.017s sys 0.001s
W20250626 19:57:36.214141 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 512dc911bfce49b29c6fc8cd90523723 because the tablet was not found. No further retry: Not found: Tablet not found: 512dc911bfce49b29c6fc8cd90523723
I20250626 19:57:36.215415 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65b16141c3d47c5b66024da59b9bbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.216590 5353 raft_consensus.cc:357] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.217370 5353 raft_consensus.cc:383] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:36.217756 5353 raft_consensus.cc:738] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:36.218983 5431 tablet_replica.cc:331] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:36.220072 5431 raft_consensus.cc:2241] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:36.221561 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7391b9eaea3e426ba4118fc8b53b72a6" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:36.221259 5353 consensus_queue.cc:260] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.222489 4973 raft_consensus.cc:3058] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:36.222616 5431 raft_consensus.cc:2270] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:36.225984 5353 ts_tablet_manager.cc:1428] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.012s user 0.001s sys 0.005s
I20250626 19:57:36.226585 5255 raft_consensus.cc:491] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:36.226889 5353 tablet_bootstrap.cc:492] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:36.227025 5255 raft_consensus.cc:513] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.228819 5255 leader_election.cc:290] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:36.231024 5431 ts_tablet_manager.cc:1905] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:36.234444 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65af5ae255d74143976392bccc50f3ba" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
W20250626 19:57:36.239084 4911 leader_election.cc:343] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:36.244325 4973 raft_consensus.cc:2466] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:36.245464 4756 leader_election.cc:304] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:36.246297 5429 raft_consensus.cc:2802] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:36.246695 5429 raft_consensus.cc:695] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:36.247584 5429 consensus_queue.cc:237] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.248581 5366 tablet_bootstrap.cc:492] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:36.248943 5366 ts_tablet_manager.cc:1397] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.004s
I20250626 19:57:36.255954 5506 leader_election.cc:290] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:36.260246 5353 tablet_bootstrap.cc:654] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:36.261324 4755 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 140774us
I20250626 19:57:36.263780 4396 catalog_manager.cc:6652] ReplaceTablet: tablet 82823fd9ceb1485386616319fa7aeb43 deleted and replaced by tablet f6ec952321924098a142f1438a283aab
I20250626 19:57:36.271968 5431 ts_tablet_manager.cc:1918] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:36.272974 5431 log.cc:1199] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/af58f34e2248432fae5c084bd3388028
I20250626 19:57:36.274076 5431 ts_tablet_manager.cc:1939] T af58f34e2248432fae5c084bd3388028 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:36.277004 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 42) took 2320 ms. Trace:
I20250626 19:57:36.279938 5431 rpcz_store.cc:276] 0626 19:57:33.956688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.956855 (+ 167us) service_pool.cc:225] Handling call
0626 19:57:36.218728 (+2261873us) ts_tablet_manager.cc:1145] Deleting tablet af58f34e2248432fae5c084bd3388028
0626 19:57:36.218819 (+ 91us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:36.260821 (+ 42002us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.271750 (+ 10929us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.276945 (+ 5195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":658,"tablet-delete.queue_time_us":2261373}
I20250626 19:57:35.909303 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c92deaa237064dee8f866e3a7a355ca4" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:36.281468 4671 raft_consensus.cc:2466] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:35.913594 5524 raft_consensus.cc:513] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:35.871676 5410 tablet_bootstrap.cc:654] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:36.283085 4677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7391b9eaea3e426ba4118fc8b53b72a6" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:36.283704 4677 raft_consensus.cc:3058] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:36.005825 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c92deaa237064dee8f866e3a7a355ca4" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:36.285621 4676 raft_consensus.cc:3058] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:36.285923 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7facb77eb579448f9533d285cba8650e" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:36.286829 4669 raft_consensus.cc:3058] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:36.290611 4677 raft_consensus.cc:2466] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:36.291952 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c92deaa237064dee8f866e3a7a355ca4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:36.067068 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a31f1a7739544706bff2a500d6026086" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:36.255658 5366 raft_consensus.cc:357] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.297016 5366 raft_consensus.cc:383] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:36.297297 5366 raft_consensus.cc:738] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:36.298192 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd6543699e254400b3516aece4486fe0" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:36.256539 4799 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a8f494f8614e7ebb8ca880b6c80e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:36.298911 4671 raft_consensus.cc:2391] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 93f885c6802341c889a40e9c20fbfff1 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:36.305840 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd6543699e254400b3516aece4486fe0" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:36.306602 4819 raft_consensus.cc:2391] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 93f885c6802341c889a40e9c20fbfff1 in this term.
I20250626 19:57:36.297824 5366 consensus_queue.cc:260] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:35.916451 4611 leader_election.cc:343] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:36.230495 4672 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65af5ae255d74143976392bccc50f3ba" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:36.319952 5433 tablet_replica.cc:331] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:36.329625 5366 ts_tablet_manager.cc:1428] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.080s user 0.006s sys 0.001s
I20250626 19:57:36.306710 4676 raft_consensus.cc:2466] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
W20250626 19:57:36.320288 4611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 404031us
W20250626 19:57:36.340554 4611 leader_election.cc:343] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:36.305614 5524 leader_election.cc:290] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:36.343719 4669 raft_consensus.cc:2466] T 7facb77eb579448f9533d285cba8650e P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:36.346037 4649 tablet_service.cc:1468] Processing CreateTablet for tablet e65b16141c3d47c5b66024da59b9bbe9 (DEFAULT_TABLE table=default.Table_bca98e7f86544ed89dbd4af5356026ca [id=d1d1bf84503e46afa724f4aa0bb60584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.334630 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:36.346546 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:36.347143 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65b16141c3d47c5b66024da59b9bbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.299612 4912 leader_election.cc:304] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
W20250626 19:57:36.349084 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 49557.9us
I20250626 19:57:36.350785 4979 raft_consensus.cc:3058] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:36.354212 4912 leader_election.cc:343] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:36.354591 4912 leader_election.cc:304] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:36.355057 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd6543699e254400b3516aece4486fe0" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:36.355405 5455 raft_consensus.cc:2747] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:36.356292 4912 leader_election.cc:304] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters: 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:36.356942 5455 raft_consensus.cc:2802] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:36.357378 5455 raft_consensus.cc:695] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:36.358151 5455 consensus_queue.cc:237] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.360093 4648 tablet_service.cc:1468] Processing CreateTablet for tablet a3636b78f4a64f9eacb4e5979768929b (DEFAULT_TABLE table=default.Table_07d4c194b22c4529b42fc16c2ebaff31 [id=c934f92aa6c9402ba2d0bae8a0288958]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.342618 5353 tablet_bootstrap.cc:492] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:36.369572 5353 ts_tablet_manager.cc:1397] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.143s user 0.014s sys 0.001s
I20250626 19:57:36.371963 5353 raft_consensus.cc:357] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:36.372946 4648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3636b78f4a64f9eacb4e5979768929b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.372898 5353 raft_consensus.cc:383] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:36.373318 5353 raft_consensus.cc:738] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:36.375478 5431 tablet_replica.cc:331] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:36.376837 5431 raft_consensus.cc:2241] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:36.377692 5431 raft_consensus.cc:2270] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:36.371265 4654 tablet_service.cc:1515] Processing DeleteTablet for tablet c92deaa237064dee8f866e3a7a355ca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:34 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:36.361219 5433 raft_consensus.cc:2241] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:36.375023 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet 82823fd9ceb1485386616319fa7aeb43 with delete_type TABLET_DATA_DELETED (replaced by tablet f6ec952321924098a142f1438a283aab) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:36.377023 4651 tablet_service.cc:1515] Processing DeleteTablet for tablet a31f1a7739544706bff2a500d6026086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:34 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:36.380139 5433 raft_consensus.cc:2270] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:36.381443 5366 tablet_bootstrap.cc:492] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:36.381872 5433 ts_tablet_manager.cc:1905] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:36.401036 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 82823fd9ceb1485386616319fa7aeb43 with delete_type TABLET_DATA_DELETED (replaced by tablet f6ec952321924098a142f1438a283aab) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:36.337327 4755 leader_election.cc:304] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:36.402786 5561 raft_consensus.cc:2747] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:57:36.402731 5353 consensus_queue.cc:260] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
W20250626 19:57:36.403574 4755 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 66374.1us
I20250626 19:57:36.405723 4755 leader_election.cc:304] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:36.408329 5550 tablet.cc:1722] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:36.409327 5561 raft_consensus.cc:2747] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:57:36.387389 4652 tablet_service.cc:1468] Processing CreateTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d (DEFAULT_TABLE table=default.Table_ff801b0f32eb4c948c82f6c4b7d42b97 [id=46243e06f82542189b78751aef158149]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.419997 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb34ffcd4da04e32a1220b3f298ec22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.397938 4651 tablet_service.cc:1468] Processing CreateTablet for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (DEFAULT_TABLE table=default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.424234 5431 ts_tablet_manager.cc:1905] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:36.432394 5255 raft_consensus.cc:2747] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:36.435153 5353 ts_tablet_manager.cc:1428] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.065s user 0.006s sys 0.000s
I20250626 19:57:36.435730 4755 leader_election.cc:304] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:36.439654 5353 tablet_bootstrap.cc:492] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
W20250626 19:57:36.442961 4395 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 69) took 2691 ms (client timeout 1000 ms). Trace:
W20250626 19:57:36.443138 4399 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 60) took 2323 ms (client timeout 1000 ms). Trace:
W20250626 19:57:36.443476 4399 rpcz_store.cc:269] 0626 19:57:34.119820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:34.157309 (+ 37489us) service_pool.cc:225] Handling call
0626 19:57:34.234198 (+ 76889us) spinlock_profiling.cc:243] Waited 72.6 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe7915557ee 00007fe78f26c624 00007fe78f26c5b2 00007fe78f26c4e1 00007fe78f26c45c 00007fe78f26b712 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:36.443076 (+2208878us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":114967,"reported_tablets":31,"spinlock_wait_cycles":166912256}
I20250626 19:57:36.446141 4950 tablet_service.cc:1515] Processing DeleteTablet for tablet ea7374ad6ba64e46b7df9852fa33f145 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:36.448253 5561 raft_consensus.cc:2802] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:36.449352 5561 raft_consensus.cc:695] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:36.451731 5561 consensus_queue.cc:237] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:36.443296 4395 rpcz_store.cc:269] 0626 19:57:33.751350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.757761 (+ 6411us) service_pool.cc:225] Handling call
0626 19:57:35.045749 (+1287988us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:35.045794 (+ 45us) write_op.cc:454] Released schema lock
0626 19:57:36.442876 (+1397082us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:35.023132 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:35.023507 (+ 375us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:35.023531 (+ 24us) write_op.cc:435] Acquired schema lock
0626 19:57:35.023544 (+ 13us) tablet.cc:660] Decoding operations
0626 19:57:35.023937 (+ 393us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:35.023976 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:35.023998 (+ 22us) tablet.cc:683] Acquiring locks for 3 operations
0626 19:57:35.024236 (+ 238us) tablet.cc:699] Row locks acquired
0626 19:57:35.024250 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:57:35.024353 (+ 103us) write_op.cc:270] Start()
0626 19:57:35.024433 (+ 80us) write_op.cc:276] Timestamp: P: 1750967855024389 usec, L: 0
0626 19:57:35.024450 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:35.025863 (+ 1413us) log.cc:844] Serialized 1284 byte log entry
0626 19:57:35.036083 (+ 10220us) op_driver.cc:464] REPLICATION: finished
0626 19:57:35.036593 (+ 510us) write_op.cc:301] APPLY: starting
0626 19:57:35.036660 (+ 67us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:35.036846 (+ 186us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:35.037587 (+ 741us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:35.037945 (+ 358us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:35.038071 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0626 19:57:35.038109 (+ 38us) write_op.cc:312] APPLY: finished
0626 19:57:35.038301 (+ 192us) log.cc:844] Serialized 35 byte log entry
0626 19:57:35.038440 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:35.038513 (+ 73us) write_op.cc:454] Released schema lock
0626 19:57:35.038544 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":14827,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":334,"apply.run_cpu_time_us":2124,"apply.run_wall_time_us":2123,"mutex_wait_us":21189,"num_ops":3,"prepare.queue_time_us":21306,"prepare.run_cpu_time_us":2347,"prepare.run_wall_time_us":10379,"raft.queue_time_us":1718,"raft.run_cpu_time_us":624,"raft.run_wall_time_us":624,"replication_time_us":11526}]]}
I20250626 19:57:36.443854 4396 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_3e24f21638574c22b00dd906bb373acb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:36.460173 4396 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e24f21638574c22b00dd906bb373acb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:36.459795 4395 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_c62b1259fcee4798bbcc9a86437cf004"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:36.461606 4395 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c62b1259fcee4798bbcc9a86437cf004 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:36.466408 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 499f6ae3262940b4859c14a53722482f (table default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]) successfully deleted
I20250626 19:57:36.472884 5255 raft_consensus.cc:491] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:36.473237 5353 tablet_bootstrap.cc:654] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:36.473343 5255 raft_consensus.cc:513] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:36.476619 5255 leader_election.cc:290] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:36.477300 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c03691d021e4ee19d5bb287ab633a3c" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:36.477947 4820 raft_consensus.cc:2466] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:36.478994 4911 leader_election.cc:304] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:36.479884 5255 raft_consensus.cc:2802] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:36.480258 5255 raft_consensus.cc:491] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:36.480607 5255 raft_consensus.cc:3058] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:36.486357 5431 ts_tablet_manager.cc:1918] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:36.488375 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (DEFAULT_TABLE table=default.Table_a5a4503fbcda4a7f95c722aaa5b4b0ce [id=760c56838ee140d78aa368a3810d19a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.489437 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.486723 5431 log.cc:1199] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/499f6ae3262940b4859c14a53722482f
W20250626 19:57:36.505625 4392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 75) took 2708 ms (client timeout 1000 ms). Trace:
I20250626 19:57:36.506549 5431 ts_tablet_manager.cc:1939] T 499f6ae3262940b4859c14a53722482f P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
W20250626 19:57:36.506640 4392 rpcz_store.cc:269] 0626 19:57:33.797030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.809339 (+ 12309us) service_pool.cc:225] Handling call
0626 19:57:36.505496 (+2696157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52642,"reported_tablets":31}
I20250626 19:57:36.508423 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 43) took 2444 ms. Trace:
I20250626 19:57:36.508723 5431 rpcz_store.cc:276] 0626 19:57:34.064367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:34.064848 (+ 481us) service_pool.cc:225] Handling call
0626 19:57:36.375333 (+2310485us) ts_tablet_manager.cc:1145] Deleting tablet 499f6ae3262940b4859c14a53722482f
0626 19:57:36.375375 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:36.465082 (+ 89707us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.486141 (+ 21059us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.508354 (+ 22213us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2309902}
I20250626 19:57:36.514035 5366 tablet_bootstrap.cc:654] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:36.487674 5255 raft_consensus.cc:513] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
W20250626 19:57:36.521701 4653 consensus_meta.cc:287] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 4d3be519a0044a2f921a503412f72e86: Time spent flushing consensus metadata: real 0.593s user 0.007s sys 0.000s
I20250626 19:57:36.512164 5353 tablet_bootstrap.cc:492] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:36.522522 5353 ts_tablet_manager.cc:1397] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.083s user 0.009s sys 0.002s
I20250626 19:57:36.526371 5353 raft_consensus.cc:357] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.529165 5353 raft_consensus.cc:383] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:36.529505 5353 raft_consensus.cc:738] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:36.530175 5353 consensus_queue.cc:260] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:36.432993 4394 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 4c0338c05a974e0f96865398722c5e8f consensus state has no leader
I20250626 19:57:36.528774 5255 leader_election.cc:290] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:36.508985 4392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_092532fc216f4744b12102be8acf3b86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:36.536562 5431 tablet_replica.cc:331] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:36.536405 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c03691d021e4ee19d5bb287ab633a3c" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:36.536955 4820 raft_consensus.cc:3058] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:36.537235 5431 raft_consensus.cc:2241] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:36.537657 5431 raft_consensus.cc:2270] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:36.534641 5353 ts_tablet_manager.cc:1428] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250626 19:57:36.540486 5353 tablet_bootstrap.cc:492] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:36.546041 5431 ts_tablet_manager.cc:1905] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:36.548251 4392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_092532fc216f4744b12102be8acf3b86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:57:36.556828 4394 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 3c03691d021e4ee19d5bb287ab633a3c (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3c03691d021e4ee19d5bb287ab633a3c consensus state has no leader
I20250626 19:57:36.568720 5353 tablet_bootstrap.cc:654] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:36.578004 4979 raft_consensus.cc:3058] T 33a75d223c274c049cf25f79fcc4b961 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:36.582564 5431 ts_tablet_manager.cc:1918] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:36.584074 5431 log.cc:1199] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/4b3bed19e7e34d24b995840a3bde5ef0
I20250626 19:57:36.585193 5431 ts_tablet_manager.cc:1939] T 4b3bed19e7e34d24b995840a3bde5ef0 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:36.441509 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet af58f34e2248432fae5c084bd3388028 (table default.Table_e25c8c1da59041118a0072aa639e3771 [id=bea62f96f1cd4c2ab35704acf7def541]) successfully deleted
W20250626 19:57:36.591789 4385 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 150363us
I20250626 19:57:36.587952 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 48) took 1716 ms. Trace:
I20250626 19:57:36.592206 5431 rpcz_store.cc:276] 0626 19:57:34.871846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:34.923251 (+ 51405us) service_pool.cc:225] Handling call
0626 19:57:36.536417 (+1613166us) ts_tablet_manager.cc:1145] Deleting tablet 4b3bed19e7e34d24b995840a3bde5ef0
0626 19:57:36.536456 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:36.572286 (+ 35830us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.582350 (+ 10064us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.587890 (+ 5540us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1612586}
I20250626 19:57:36.522145 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c03691d021e4ee19d5bb287ab633a3c" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:36.523658 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet ea7374ad6ba64e46b7df9852fa33f145 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:36.596231 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet 612f87796cfa44ddbb19b02797031db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:25 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:36.593662 5431 tablet_replica.cc:331] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:36.597512 4950 tablet_service.cc:1468] Processing CreateTablet for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (DEFAULT_TABLE table=default.Table_a5a4503fbcda4a7f95c722aaa5b4b0ce [id=760c56838ee140d78aa368a3810d19a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.598300 5431 raft_consensus.cc:2241] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:36.525035 4654 tablet_service.cc:1515] Processing DeleteTablet for tablet afd1fafb53434957b002fad5c77431d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:36.525974 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 5785508e37624f02bc2f6cc93007e3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:28 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:36.528507 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c03691d021e4ee19d5bb287ab633a3c" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:36.533839 4650 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a8f494f8614e7ebb8ca880b6c80e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:36.539608 4651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 465ed0fe56fb4960a58e92b9b3d594e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.598712 4950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.599175 4952 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e453a4f8ca44f4a940928d9f4fcd5be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:31 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:36.600071 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet afd1fafb53434957b002fad5c77431d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:36.598719 5431 raft_consensus.cc:2270] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250626 19:57:36.600450 4912 leader_election.cc:343] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:36.606562 5431 ts_tablet_manager.cc:1905] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:36.598940 4956 tablet_service.cc:1515] Processing DeleteTablet for tablet 5785508e37624f02bc2f6cc93007e3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:28 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:36.607784 5438 ts_tablet_manager.cc:1918] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:57:36.612478 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 499f6ae3262940b4859c14a53722482f (table default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]) successfully deleted
I20250626 19:57:36.613653 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 4b3bed19e7e34d24b995840a3bde5ef0 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]) successfully deleted
I20250626 19:57:36.614946 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d8ffa9b658c413fa06d02ee6635dd1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:31 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:36.616212 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet 04b0f44c1b564fdf98408881678a90a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c03691d021e4ee19d5bb287ab633a3c) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:36.617604 4956 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a8f494f8614e7ebb8ca880b6c80e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:36.612005 4820 raft_consensus.cc:2466] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:36.620873 5438 log.cc:1199] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/5785508e37624f02bc2f6cc93007e3f8
I20250626 19:57:36.626547 5438 ts_tablet_manager.cc:1939] T 5785508e37624f02bc2f6cc93007e3f8 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:36.628692 4956 tablet_service.cc:1515] Processing DeleteTablet for tablet af58f34e2248432fae5c084bd3388028 with delete_type TABLET_DATA_DELETED (replaced by tablet e60b4d53c40c4dffab928f0e3fb4a80b) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:36.630128 4911 leader_election.cc:304] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:36.630896 5255 raft_consensus.cc:2802] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:36.631345 5255 raft_consensus.cc:695] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:36.632102 5255 consensus_queue.cc:237] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:36.551534 5410 tablet_bootstrap.cc:492] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:36.634410 5410 ts_tablet_manager.cc:1397] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.970s user 0.012s sys 0.005s
I20250626 19:57:36.633215 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 512dc911bfce49b29c6fc8cd90523723 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:26 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:36.639679 5431 ts_tablet_manager.cc:1918] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:36.640348 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet af58f34e2248432fae5c084bd3388028 with delete_type TABLET_DATA_DELETED (replaced by tablet e60b4d53c40c4dffab928f0e3fb4a80b) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:36.642113 5431 log.cc:1199] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/fd6543699e254400b3516aece4486fe0
I20250626 19:57:36.643827 5431 ts_tablet_manager.cc:1939] T fd6543699e254400b3516aece4486fe0 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:36.648360 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 47) took 2068 ms. Trace:
I20250626 19:57:36.649191 5431 rpcz_store.cc:276] 0626 19:57:34.648145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:34.901612 (+253467us) service_pool.cc:225] Handling call
0626 19:57:36.593522 (+1691910us) ts_tablet_manager.cc:1145] Deleting tablet fd6543699e254400b3516aece4486fe0
0626 19:57:36.593560 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:36.617738 (+ 24178us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.639480 (+ 21742us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.648297 (+ 8817us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5,"spinlock_wait_cycles":3328,"tablet-delete.queue_time_us":1589082}
I20250626 19:57:36.652228 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet fd6543699e254400b3516aece4486fe0 (table default.Table_fde5b6bd4ee8417e85ffccad3bd2a581 [id=ab0bf254569748ba8d31931d4bfb105d]) successfully deleted
I20250626 19:57:36.653327 5431 tablet_replica.cc:331] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:36.654582 5431 raft_consensus.cc:2241] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:36.654314 5410 raft_consensus.cc:357] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:36.655020 5410 raft_consensus.cc:383] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:36.655308 5410 raft_consensus.cc:738] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:36.656216 5431 raft_consensus.cc:2270] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:36.655922 5410 consensus_queue.cc:260] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:36.657821 5431 ts_tablet_manager.cc:1905] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:36.661012 4912 leader_election.cc:343] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:36.662218 5255 raft_consensus.cc:491] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:36.662818 5255 raft_consensus.cc:513] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.667575 5255 leader_election.cc:290] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:36.668517 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61fcd55f6d7b4a369383d4f73d5f483d" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:36.672528 4820 raft_consensus.cc:2466] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:36.675496 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61fcd55f6d7b4a369383d4f73d5f483d" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:36.676167 4911 leader_election.cc:304] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
W20250626 19:57:36.677551 4912 leader_election.cc:343] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:36.677940 5569 raft_consensus.cc:2802] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:36.678282 5569 raft_consensus.cc:491] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:36.678325 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d8ffa9b658c413fa06d02ee6635dd1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:31 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:36.678602 5569 raft_consensus.cc:3058] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:36.670209 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 29) took 2966 ms. Trace:
I20250626 19:57:36.687168 5569 raft_consensus.cc:513] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.689731 5438 rpcz_store.cc:276] 0626 19:57:33.703819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.706270 (+ 2451us) service_pool.cc:225] Handling call
0626 19:57:36.122421 (+2416151us) ts_tablet_manager.cc:1145] Deleting tablet 5785508e37624f02bc2f6cc93007e3f8
0626 19:57:36.122466 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:36.344070 (+221604us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.607561 (+263491us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.670153 (+ 62592us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2415527}
I20250626 19:57:36.691601 5569 leader_election.cc:290] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:36.693998 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 5785508e37624f02bc2f6cc93007e3f8 (table default.Table_b401c8a5ab6744029f693b1b5e0f306d [id=95832380520e4cf18d762654fb684db3]) successfully deleted
I20250626 19:57:36.672127 5366 tablet_bootstrap.cc:492] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:36.696023 5366 ts_tablet_manager.cc:1397] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.315s user 0.008s sys 0.004s
I20250626 19:57:36.692967 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61fcd55f6d7b4a369383d4f73d5f483d" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:36.698966 4820 raft_consensus.cc:3058] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:36.705179 5431 ts_tablet_manager.cc:1918] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:57:36.705698 5431 log.cc:1199] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/1cb6bda529624ac5b5a87a03b4a02d56
I20250626 19:57:36.708842 5431 ts_tablet_manager.cc:1939] T 1cb6bda529624ac5b5a87a03b4a02d56 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:36.709157 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e453a4f8ca44f4a940928d9f4fcd5be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:31 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:36.709563 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 33) took 3000 ms. Trace:
I20250626 19:57:36.709951 5438 rpcz_store.cc:276] 0626 19:57:33.709498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.712649 (+ 3151us) service_pool.cc:225] Handling call
0626 19:57:36.701375 (+2988726us) ts_tablet_manager.cc:1145] Deleting tablet 3bd8dae5699c4725915814044d76a257
0626 19:57:36.701423 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:36.709498 (+ 8075us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2988192}
I20250626 19:57:36.714329 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 54) took 1622 ms. Trace:
I20250626 19:57:36.714694 5431 rpcz_store.cc:276] 0626 19:57:35.091501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:35.103967 (+ 12466us) service_pool.cc:225] Handling call
0626 19:57:36.653164 (+1549197us) ts_tablet_manager.cc:1145] Deleting tablet 1cb6bda529624ac5b5a87a03b4a02d56
0626 19:57:36.653207 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:36.674401 (+ 21194us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.704954 (+ 30553us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.714244 (+ 9290us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1548470}
I20250626 19:57:36.716279 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 1cb6bda529624ac5b5a87a03b4a02d56 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]) successfully deleted
I20250626 19:57:36.719628 5410 ts_tablet_manager.cc:1428] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.085s user 0.006s sys 0.000s
I20250626 19:57:36.720465 5431 tablet_replica.cc:331] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:36.721750 5431 raft_consensus.cc:2241] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:36.723409 5431 pending_rounds.cc:70] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6: Trying to abort 1 pending ops.
I20250626 19:57:36.723053 5571 tablet.cc:1722] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:36.724567 5431 pending_rounds.cc:77] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964339662098432 op_type: NO_OP noop_request { }
I20250626 19:57:36.725626 5431 raft_consensus.cc:2887] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:57:36.726302 5431 raft_consensus.cc:2270] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:36.729091 5431 ts_tablet_manager.cc:1905] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:36.737344 4397 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 2ab0320af47049cca3ca87172b9fe5e5
I20250626 19:57:36.740120 4397 catalog_manager.cc:6652] ReplaceTablet: tablet 2ab0320af47049cca3ca87172b9fe5e5 deleted and replaced by tablet 609d36d46bb84017b2a3a7ad48bfccfc
I20250626 19:57:36.740640 4397 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34796 (request call id 95) took 1545 ms. Trace:
I20250626 19:57:36.742280 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61fcd55f6d7b4a369383d4f73d5f483d" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
W20250626 19:57:36.743811 4912 leader_election.cc:343] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:36.747769 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 3bd8dae5699c4725915814044d76a257 because the tablet was not found. No further retry: Not found: Tablet not found: 3bd8dae5699c4725915814044d76a257
I20250626 19:57:36.698550 5366 raft_consensus.cc:357] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.740579 4800 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ab0320af47049cca3ca87172b9fe5e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 609d36d46bb84017b2a3a7ad48bfccfc) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:36.748545 5366 raft_consensus.cc:383] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:36.748813 5366 raft_consensus.cc:738] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:36.749261 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 34) took 3012 ms. Trace:
I20250626 19:57:36.749544 5438 rpcz_store.cc:276] 0626 19:57:33.734403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.734553 (+ 150us) service_pool.cc:225] Handling call
0626 19:57:36.746193 (+3011640us) ts_tablet_manager.cc:1145] Deleting tablet 04b0f44c1b564fdf98408881678a90a1
0626 19:57:36.746234 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:36.746398 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3011070}
W20250626 19:57:36.750499 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 04b0f44c1b564fdf98408881678a90a1 because the tablet was not found. No further retry: Not found: Tablet not found: 04b0f44c1b564fdf98408881678a90a1
I20250626 19:57:36.743235 4395 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 99) took 1294 ms. Trace:
I20250626 19:57:36.751307 5366 consensus_queue.cc:260] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.752401 5438 tablet_replica.cc:331] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:36.753082 5438 raft_consensus.cc:2241] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:36.757719 4390 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 96) took 1530 ms. Trace:
I20250626 19:57:36.763157 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ab0320af47049cca3ca87172b9fe5e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 609d36d46bb84017b2a3a7ad48bfccfc) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:36.764473 5438 raft_consensus.cc:2270] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:36.766343 5438 ts_tablet_manager.cc:1905] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:36.767726 5366 ts_tablet_manager.cc:1428] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.071s user 0.005s sys 0.004s
I20250626 19:57:36.769645 5431 ts_tablet_manager.cc:1918] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:36.770471 5410 tablet_bootstrap.cc:492] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:36.771651 5431 log.cc:1199] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/c92deaa237064dee8f866e3a7a355ca4
I20250626 19:57:36.772295 5366 tablet_bootstrap.cc:492] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:36.773030 5431 ts_tablet_manager.cc:1939] T c92deaa237064dee8f866e3a7a355ca4 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
W20250626 19:57:36.773278 4394 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 76) took 1904 ms (client timeout 1000 ms). Trace:
I20250626 19:57:36.776149 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet c92deaa237064dee8f866e3a7a355ca4 (table default.Table_a2a4a2c9c59248dfb73f7db218b0d982 [id=0d00dcf2b9f64037b166b83f38e1a72a]) successfully deleted
W20250626 19:57:36.780104 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 82823fd9ceb1485386616319fa7aeb43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82823fd9ceb1485386616319fa7aeb43 already in progress: creating tablet
I20250626 19:57:36.781672 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ab0320af47049cca3ca87172b9fe5e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 609d36d46bb84017b2a3a7ad48bfccfc) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:36.781934 4649 tablet_service.cc:1515] Processing DeleteTablet for tablet 499f6ae3262940b4859c14a53722482f with delete_type TABLET_DATA_DELETED (replaced by tablet 6b8473259485460783f62f8193d5b02c) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:36.782270 5431 tablet_replica.cc:331] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:36.783336 5431 raft_consensus.cc:2241] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:36.783336 4653 tablet_service.cc:1468] Processing CreateTablet for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (DEFAULT_TABLE table=default.Table_a5a4503fbcda4a7f95c722aaa5b4b0ce [id=760c56838ee140d78aa368a3810d19a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:36.784621 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:36.784665 5431 pending_rounds.cc:70] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6: Trying to abort 1 pending ops.
I20250626 19:57:36.785230 5431 pending_rounds.cc:77] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964330950819840 op_type: NO_OP noop_request { }
I20250626 19:57:36.785915 5431 raft_consensus.cc:2887] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:57:36.786612 5431 raft_consensus.cc:2270] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:36.742106 4820 raft_consensus.cc:2466] T 61fcd55f6d7b4a369383d4f73d5f483d P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:36.793870 4911 leader_election.cc:304] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:36.795213 5569 raft_consensus.cc:2802] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:36.795612 5569 raft_consensus.cc:695] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:36.796092 5524 consensus_queue.cc:1035] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:36.796370 5569 consensus_queue.cc:237] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.808063 5410 tablet_bootstrap.cc:654] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:36.817687 4611 leader_election.cc:343] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:36.818290 4611 leader_election.cc:304] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
W20250626 19:57:36.819902 4611 raft_consensus.cc:2684] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250626 19:57:36.821158 5366 tablet_bootstrap.cc:654] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:36.751140 4395 rpcz_store.cc:276] 0626 19:57:35.448885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.459390 (+1010505us) service_pool.cc:225] Handling call
0626 19:57:36.462093 (+ 2703us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:36.464248 (+ 2155us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:36.569772 (+105524us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:36.742922 (+173150us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:36.743179 (+ 257us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:36.465420 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:36.465768 (+ 348us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:36.465791 (+ 23us) write_op.cc:435] Acquired schema lock
0626 19:57:36.465803 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:36.466047 (+ 244us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:36.466079 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:36.466126 (+ 47us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:36.466313 (+ 187us) tablet.cc:699] Row locks acquired
0626 19:57:36.466325 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:36.466427 (+ 102us) write_op.cc:270] Start()
0626 19:57:36.466505 (+ 78us) write_op.cc:276] Timestamp: P: 1750967856465617 usec, L: 0
0626 19:57:36.466520 (+ 15us) op_driver.cc:348] REPLICATION: starting
0626 19:57:36.468379 (+ 1859us) log.cc:844] Serialized 433 byte log entry
0626 19:57:36.565678 (+ 97299us) op_driver.cc:464] REPLICATION: finished
0626 19:57:36.568300 (+ 2622us) write_op.cc:301] APPLY: starting
0626 19:57:36.568380 (+ 80us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:36.568482 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:36.568576 (+ 94us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:36.568980 (+ 404us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:36.569120 (+ 140us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:36.569159 (+ 39us) write_op.cc:312] APPLY: finished
0626 19:57:36.569346 (+ 187us) log.cc:844] Serialized 29 byte log entry
0626 19:57:36.569479 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:36.569541 (+ 62us) write_op.cc:454] Released schema lock
0626 19:57:36.569570 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2487,"apply.run_cpu_time_us":1456,"apply.run_wall_time_us":1455,"mutex_wait_us":20964,"num_ops":2,"prepare.queue_time_us":91,"prepare.run_cpu_time_us":1964,"prepare.run_wall_time_us":24274,"raft.queue_time_us":22988,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":15,"replication_time_us":99079,"thread_start_us":146251,"threads_started":1}]]}
W20250626 19:57:36.835722 4398 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 61) took 1467 ms (client timeout 1000 ms). Trace:
W20250626 19:57:36.836419 4398 rpcz_store.cc:269] 0626 19:57:35.137025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:35.152099 (+ 15074us) service_pool.cc:225] Handling call
0626 19:57:36.604257 (+1452158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":822,"reported_tablets":41}
I20250626 19:57:36.839458 4398 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_8ed57e01e9b54a829b11bc477378d778" } new_table_name: "default.Table_573c94e619134cddab6aac4f11c37871" modify_external_catalogs: true
I20250626 19:57:36.758075 4390 rpcz_store.cc:276] 0626 19:57:35.227671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.042581 (+814910us) service_pool.cc:225] Handling call
0626 19:57:36.048698 (+ 6117us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:36.050293 (+ 1595us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:36.571840 (+521547us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:36.757340 (+185500us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:36.757652 (+ 312us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:36.441133 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:36.441514 (+ 381us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:36.441535 (+ 21us) write_op.cc:435] Acquired schema lock
0626 19:57:36.441553 (+ 18us) tablet.cc:660] Decoding operations
0626 19:57:36.441814 (+ 261us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:36.441850 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:36.441870 (+ 20us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:36.442045 (+ 175us) tablet.cc:699] Row locks acquired
0626 19:57:36.442058 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:57:36.442173 (+ 115us) write_op.cc:270] Start()
0626 19:57:36.442249 (+ 76us) write_op.cc:276] Timestamp: P: 1750967856441355 usec, L: 0
0626 19:57:36.442263 (+ 14us) op_driver.cc:348] REPLICATION: starting
0626 19:57:36.442702 (+ 439us) log.cc:844] Serialized 433 byte log entry
0626 19:57:36.565411 (+122709us) op_driver.cc:464] REPLICATION: finished
0626 19:57:36.566226 (+ 815us) write_op.cc:301] APPLY: starting
0626 19:57:36.566296 (+ 70us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:36.566425 (+ 129us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:36.566581 (+ 156us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:36.567097 (+ 516us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:36.567249 (+ 152us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:36.567304 (+ 55us) write_op.cc:312] APPLY: finished
0626 19:57:36.567518 (+ 214us) log.cc:844] Serialized 29 byte log entry
0626 19:57:36.567697 (+ 179us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:36.567779 (+ 82us) write_op.cc:454] Released schema lock
0626 19:57:36.567809 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:36.568048 (+ 239us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:36.568082 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":644,"apply.run_cpu_time_us":1788,"apply.run_wall_time_us":1788,"num_ops":2,"prepare.queue_time_us":96,"prepare.run_cpu_time_us":1877,"prepare.run_wall_time_us":2048,"raft.queue_time_us":97,"raft.run_cpu_time_us":16,"raft.run_wall_time_us":16,"replication_time_us":123033}]]}
I20250626 19:57:36.842875 4390 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_ff801b0f32eb4c948c82f6c4b7d42b97" } modify_external_catalogs: true
I20250626 19:57:36.741035 4397 rpcz_store.cc:276] 0626 19:57:35.195585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.036113 (+840528us) service_pool.cc:225] Handling call
0626 19:57:36.740558 (+704445us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:36.409672 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:36.410045 (+ 373us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:36.410067 (+ 22us) write_op.cc:435] Acquired schema lock
0626 19:57:36.410080 (+ 13us) tablet.cc:660] Decoding operations
0626 19:57:36.410359 (+ 279us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:36.410433 (+ 74us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:36.410462 (+ 29us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:36.415168 (+ 4706us) tablet.cc:699] Row locks acquired
0626 19:57:36.415182 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:57:36.415285 (+ 103us) write_op.cc:270] Start()
0626 19:57:36.415368 (+ 83us) write_op.cc:276] Timestamp: P: 1750967856415444 usec, L: 0
0626 19:57:36.415383 (+ 15us) op_driver.cc:348] REPLICATION: starting
0626 19:57:36.423277 (+ 7894us) log.cc:844] Serialized 659 byte log entry
0626 19:57:36.465572 (+ 42295us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b1800041380. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe791479af1 00007fe791486441 00007fe78885b60f 00007fe78885d8b1 00007fe78885d869 00007fe78885d7f9 00007fe78885d7c1 00007fe78885cabd 000000000059bcd4 000000000059bb09 00007fe78883e636
0626 19:57:36.487199 (+ 21627us) op_driver.cc:464] REPLICATION: finished
0626 19:57:36.487637 (+ 438us) write_op.cc:301] APPLY: starting
0626 19:57:36.487694 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:36.487796 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:36.487813 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:36.488184 (+ 371us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:36.488396 (+ 212us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:36.488443 (+ 47us) write_op.cc:312] APPLY: finished
0626 19:57:36.488616 (+ 173us) log.cc:844] Serialized 29 byte log entry
0626 19:57:36.488762 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:36.488829 (+ 67us) write_op.cc:454] Released schema lock
0626 19:57:36.488863 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:36.489072 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:36.489094 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":271,"apply.run_cpu_time_us":1413,"apply.run_wall_time_us":1413,"mutex_wait_us":23802,"num_ops":2,"prepare.queue_time_us":97,"prepare.run_cpu_time_us":2126,"prepare.run_wall_time_us":13904,"raft.queue_time_us":11019,"raft.run_cpu_time_us":813,"raft.run_wall_time_us":14754,"replication_time_us":71707,"spinlock_wait_cycles":52861824}]]}
W20250626 19:57:36.876538 4611 consensus_peers.cc:487] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:36.879297 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 85523us
I20250626 19:57:36.879932 5431 ts_tablet_manager.cc:1905] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:36.899746 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 46218.1us
I20250626 19:57:36.904531 5571 tablet.cc:1722] T 5e20ea7c17c34fbd9c117275e3c14466 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250626 19:57:36.920135 4611 consensus_peers.cc:487] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:36.920696 5353 tablet_bootstrap.cc:492] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:36.921108 5353 ts_tablet_manager.cc:1397] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.381s user 0.052s sys 0.056s
I20250626 19:57:36.922245 5592 tablet.cc:1722] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:36.928983 5410 tablet_bootstrap.cc:492] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:36.847784 4397 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_fbd8a487fac74f3087145875222b3dab" } new_table_name: "default.Table_35dcbe7e914846d8b6b52efa647c4d98" modify_external_catalogs: true
I20250626 19:57:36.927924 5366 tablet_bootstrap.cc:492] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:36.947808 5366 ts_tablet_manager.cc:1397] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.176s user 0.003s sys 0.009s
I20250626 19:57:36.949764 5410 ts_tablet_manager.cc:1397] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.180s user 0.011s sys 0.002s
I20250626 19:57:36.950286 5366 raft_consensus.cc:357] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.950985 5366 raft_consensus.cc:383] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250626 19:57:36.773612 4394 rpcz_store.cc:269] 0626 19:57:34.868887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:34.868998 (+ 111us) service_pool.cc:225] Handling call
0626 19:57:35.658873 (+789875us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:35.658911 (+ 38us) write_op.cc:454] Released schema lock
0626 19:57:36.773208 (+1114297us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:35.640264 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:35.640623 (+ 359us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:35.640647 (+ 24us) write_op.cc:435] Acquired schema lock
0626 19:57:35.640659 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:35.640962 (+ 303us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:35.641000 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:35.641021 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:35.641245 (+ 224us) tablet.cc:699] Row locks acquired
0626 19:57:35.641258 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:57:35.641361 (+ 103us) write_op.cc:270] Start()
0626 19:57:35.641437 (+ 76us) write_op.cc:276] Timestamp: P: 1750967855641437 usec, L: 0
0626 19:57:35.641451 (+ 14us) op_driver.cc:348] REPLICATION: starting
0626 19:57:35.641960 (+ 509us) log.cc:844] Serialized 907 byte log entry
0626 19:57:35.651550 (+ 9590us) op_driver.cc:464] REPLICATION: finished
0626 19:57:35.651959 (+ 409us) write_op.cc:301] APPLY: starting
0626 19:57:35.652009 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:35.652107 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:35.652197 (+ 90us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:35.652414 (+ 217us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:35.652549 (+ 135us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:35.652591 (+ 42us) write_op.cc:312] APPLY: finished
0626 19:57:35.652771 (+ 180us) log.cc:844] Serialized 29 byte log entry
0626 19:57:35.652908 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:35.652974 (+ 66us) write_op.cc:454] Released schema lock
0626 19:57:35.653019 (+ 45us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0626 19:57:35.744800 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:35.745236 (+ 436us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:35.745262 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:35.745274 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:35.745522 (+ 248us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:35.745562 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:35.745584 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:35.745742 (+ 158us) tablet.cc:699] Row locks acquired
0626 19:57:35.745754 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:35.745862 (+ 108us) write_op.cc:270] Start()
0626 19:57:35.745939 (+ 77us) write_op.cc:276] Timestamp: P: 1750967855745977 usec, L: 0
0626 19:57:35.745956 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:35.746469 (+ 513us) log.cc:844] Serialized 356 byte log entry
0626 19:57:35.915638 (+169169us) op_driver.cc:464] REPLICATION: finished
0626 19:57:35.915931 (+ 293us) write_op.cc:301] APPLY: starting
0626 19:57:35.915983 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:35.916070 (+ 87us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:35.916158 (+ 88us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:35.916346 (+ 188us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:35.916485 (+ 139us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:35.916525 (+ 40us) write_op.cc:312] APPLY: finished
0626 19:57:35.916697 (+ 172us) log.cc:844] Serialized 23 byte log entry
0626 19:57:35.916826 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:35.916885 (+ 59us) write_op.cc:454] Released schema lock
0626 19:57:35.916916 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:35.917157 (+ 241us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:35.917179 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":175188,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":275,"apply.run_cpu_time_us":1219,"apply.run_wall_time_us":1218,"num_ops":2,"prepare.queue_time_us":14859,"prepare.run_cpu_time_us":2002,"prepare.run_wall_time_us":2001,"raft.queue_time_us":3318,"raft.run_cpu_time_us":727,"raft.run_wall_time_us":726,"replication_time_us":10005}],["op",{"apply.queue_time_us":155,"apply.run_cpu_time_us":1180,"apply.run_wall_time_us":1178,"num_ops":1,"prepare.queue_time_us":62695,"prepare.run_cpu_time_us":1991,"prepare.run_wall_time_us":1991,"raft.queue_time_us":1130,"raft.run_cpu_time_us":631,"raft.run_wall_time_us":630,"replication_time_us":169576}]]}
I20250626 19:57:36.951296 5366 raft_consensus.cc:738] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:36.951843 4394 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_beec4e8c60ae4e6880c0ff709d3c0ff3" } modify_external_catalogs: true
I20250626 19:57:36.952443 5366 consensus_queue.cc:260] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.953799 5597 tablet.cc:1722] T 5e20ea7c17c34fbd9c117275e3c14466 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250626 19:57:36.924978 5353 raft_consensus.cc:357] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.910364 5438 ts_tablet_manager.cc:1918] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:36.971448 5353 raft_consensus.cc:383] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:36.971629 5438 log.cc:1199] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/af58f34e2248432fae5c084bd3388028
I20250626 19:57:36.971752 5353 raft_consensus.cc:738] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:36.972527 5438 ts_tablet_manager.cc:1939] T af58f34e2248432fae5c084bd3388028 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:36.972378 5353 consensus_queue.cc:260] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.973560 5366 ts_tablet_manager.cc:1428] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.025s user 0.004s sys 0.001s
I20250626 19:57:36.974498 5353 ts_tablet_manager.cc:1428] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.051s user 0.004s sys 0.001s
I20250626 19:57:36.954018 5431 ts_tablet_manager.cc:1918] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:36.975044 5431 log.cc:1199] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/a31f1a7739544706bff2a500d6026086
I20250626 19:57:36.975739 5431 ts_tablet_manager.cc:1939] T a31f1a7739544706bff2a500d6026086 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:36.979900 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet a31f1a7739544706bff2a500d6026086 (table default.Table_fc5a0fda8b274c989cb3b9189b6a838d [id=ee7713e5a64e4b5b8e32bc5862786d05]) successfully deleted
W20250626 19:57:36.980670 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet ea7374ad6ba64e46b7df9852fa33f145 because the tablet was not found. No further retry: Not found: Tablet not found: ea7374ad6ba64e46b7df9852fa33f145
W20250626 19:57:36.981340 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 612f87796cfa44ddbb19b02797031db9 because the tablet was not found. No further retry: Not found: Tablet not found: 612f87796cfa44ddbb19b02797031db9
W20250626 19:57:36.987687 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 1e453a4f8ca44f4a940928d9f4fcd5be because the tablet was not found. No further retry: Not found: Tablet not found: 1e453a4f8ca44f4a940928d9f4fcd5be
I20250626 19:57:36.988777 5353 tablet_bootstrap.cc:492] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:36.989355 5410 raft_consensus.cc:357] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.990152 5410 raft_consensus.cc:383] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:36.990689 5410 raft_consensus.cc:738] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:36.992004 5410 consensus_queue.cc:260] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:36.994402 5353 tablet_bootstrap.cc:654] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:36.995373 4396 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 98) took 1637 ms. Trace:
I20250626 19:57:36.998926 4394 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_beec4e8c60ae4e6880c0ff709d3c0ff3" } modify_external_catalogs: true
I20250626 19:57:37.001178 4392 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 100) took 1466 ms. Trace:
I20250626 19:57:37.000244 4390 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_ff801b0f32eb4c948c82f6c4b7d42b97" } modify_external_catalogs: true
I20250626 19:57:36.995687 4396 rpcz_store.cc:276] 0626 19:57:35.357548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.409994 (+1052446us) service_pool.cc:225] Handling call
0626 19:57:36.460565 (+ 50571us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:36.461958 (+ 1393us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:36.817428 (+355470us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:36.995009 (+177581us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:36.995313 (+ 304us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:36.489802 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:36.490133 (+ 331us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:36.490155 (+ 22us) write_op.cc:435] Acquired schema lock
0626 19:57:36.490167 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:36.490429 (+ 262us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:36.490465 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:36.490485 (+ 20us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:36.490662 (+ 177us) tablet.cc:699] Row locks acquired
0626 19:57:36.490676 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:57:36.490800 (+ 124us) write_op.cc:270] Start()
0626 19:57:36.490875 (+ 75us) write_op.cc:276] Timestamp: P: 1750967856489977 usec, L: 0
0626 19:57:36.490891 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:57:36.492221 (+ 1330us) log.cc:844] Serialized 433 byte log entry
0626 19:57:36.791853 (+299632us) op_driver.cc:464] REPLICATION: finished
0626 19:57:36.815004 (+ 23151us) write_op.cc:301] APPLY: starting
0626 19:57:36.815074 (+ 70us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:36.815213 (+ 139us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:36.815367 (+ 154us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:36.815807 (+ 440us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:36.816006 (+ 199us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:36.816070 (+ 64us) write_op.cc:312] APPLY: finished
0626 19:57:36.816295 (+ 225us) log.cc:844] Serialized 29 byte log entry
0626 19:57:36.817041 (+ 746us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:36.817129 (+ 88us) write_op.cc:454] Released schema lock
0626 19:57:36.817174 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:36.818012 (+ 838us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:36.818074 (+ 62us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":22927,"apply.run_cpu_time_us":1934,"apply.run_wall_time_us":2934,"num_ops":2,"prepare.queue_time_us":12283,"prepare.run_cpu_time_us":1915,"prepare.run_wall_time_us":2669,"raft.queue_time_us":161,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":14,"replication_time_us":299659,"spinlock_wait_cycles":1941248}]]}
I20250626 19:57:37.011826 4396 master_service.cc:948] ReplaceTablet: received request to replace tablet 36997dab0ab3430bba63a90805759b30 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:37.013332 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 5785508e37624f02bc2f6cc93007e3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:28 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:37.015445 5366 tablet_bootstrap.cc:492] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:37.001520 4392 rpcz_store.cc:276] 0626 19:57:35.535151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.508271 (+973120us) service_pool.cc:225] Handling call
0626 19:57:36.763964 (+255693us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:36.864130 (+100166us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:36.948732 (+ 84602us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:37.000799 (+ 52067us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:37.001108 (+ 309us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:36.910211 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:36.910586 (+ 375us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:36.910660 (+ 74us) write_op.cc:435] Acquired schema lock
0626 19:57:36.910674 (+ 14us) tablet.cc:660] Decoding operations
0626 19:57:36.910972 (+ 298us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:36.911009 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:36.911031 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:36.911226 (+ 195us) tablet.cc:699] Row locks acquired
0626 19:57:36.911239 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:57:36.911344 (+ 105us) write_op.cc:270] Start()
0626 19:57:36.911424 (+ 80us) write_op.cc:276] Timestamp: P: 1750967856910710 usec, L: 0
0626 19:57:36.911440 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:57:36.911907 (+ 467us) log.cc:844] Serialized 433 byte log entry
0626 19:57:36.939730 (+ 27823us) op_driver.cc:464] REPLICATION: finished
0626 19:57:36.940103 (+ 373us) write_op.cc:301] APPLY: starting
0626 19:57:36.940171 (+ 68us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:36.940274 (+ 103us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:36.940369 (+ 95us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:36.940730 (+ 361us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:36.941365 (+ 635us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:36.941439 (+ 74us) write_op.cc:312] APPLY: finished
0626 19:57:36.941683 (+ 244us) log.cc:844] Serialized 29 byte log entry
0626 19:57:36.942395 (+ 712us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:36.942459 (+ 64us) write_op.cc:454] Released schema lock
0626 19:57:36.942487 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:36.942687 (+ 200us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:36.942712 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":209,"apply.run_cpu_time_us":1612,"apply.run_wall_time_us":2554,"num_ops":2,"prepare.queue_time_us":29475,"prepare.run_cpu_time_us":1901,"prepare.run_wall_time_us":1900,"replication_time_us":28185}]]}
I20250626 19:57:36.976403 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 35) took 3158 ms. Trace:
I20250626 19:57:37.017318 5438 rpcz_store.cc:276] 0626 19:57:33.818042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:33.819987 (+ 1945us) service_pool.cc:225] Handling call
0626 19:57:36.752261 (+2932274us) ts_tablet_manager.cc:1145] Deleting tablet af58f34e2248432fae5c084bd3388028
0626 19:57:36.752300 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:36.788630 (+ 36330us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.910146 (+121516us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:36.976348 (+ 66202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3719,"tablet-delete.queue_time_us":2931672}
I20250626 19:57:37.018725 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet af58f34e2248432fae5c084bd3388028 (table default.Table_e25c8c1da59041118a0072aa639e3771 [id=bea62f96f1cd4c2ab35704acf7def541]) successfully deleted
I20250626 19:57:37.019815 5438 tablet_replica.cc:331] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:37.020615 5438 raft_consensus.cc:2241] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:37.021205 5438 raft_consensus.cc:2270] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:37.022517 5366 tablet_bootstrap.cc:654] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:37.023110 5438 ts_tablet_manager.cc:1905] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:37.026829 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet afd1fafb53434957b002fad5c77431d4 because the tablet was not found. No further retry: Not found: Tablet not found: afd1fafb53434957b002fad5c77431d4
I20250626 19:57:37.028270 4392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_38eec4fd44e848378efabe2ebbc88983"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:37.029963 4392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38eec4fd44e848378efabe2ebbc88983 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:37.037449 5431 tablet_replica.cc:331] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:37.039755 5431 raft_consensus.cc:2241] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:37.040330 5431 raft_consensus.cc:2270] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 19:57:37.040865 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 5785508e37624f02bc2f6cc93007e3f8 because the tablet was not found. No further retry: Not found: Tablet not found: 5785508e37624f02bc2f6cc93007e3f8
W20250626 19:57:37.041567 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 4d8ffa9b658c413fa06d02ee6635dd1b because the tablet was not found. No further retry: Not found: Tablet not found: 4d8ffa9b658c413fa06d02ee6635dd1b
W20250626 19:57:37.042131 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 04b0f44c1b564fdf98408881678a90a1 because the tablet was not found. No further retry: Not found: Tablet not found: 04b0f44c1b564fdf98408881678a90a1
I20250626 19:57:37.042948 5410 ts_tablet_manager.cc:1428] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.093s user 0.000s sys 0.007s
W20250626 19:57:37.043057 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 78a8f494f8614e7ebb8ca880b6c80e78 because the tablet was not found. No further retry: Not found: Tablet not found: 78a8f494f8614e7ebb8ca880b6c80e78
W20250626 19:57:37.043924 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet af58f34e2248432fae5c084bd3388028 because the tablet was not found. No further retry: Not found: Tablet not found: af58f34e2248432fae5c084bd3388028
I20250626 19:57:37.045022 5366 tablet_bootstrap.cc:492] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:37.045464 5366 ts_tablet_manager.cc:1397] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.008s
I20250626 19:57:37.046154 5431 ts_tablet_manager.cc:1905] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:37.058301 5410 tablet_bootstrap.cc:492] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:37.062704 5366 raft_consensus.cc:357] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.075598 5561 consensus_queue.cc:1035] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:37.084996 5429 consensus_queue.cc:1035] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:37.103844 5592 tablet.cc:1722] T 5e20ea7c17c34fbd9c117275e3c14466 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250626 19:57:37.103333 5438 ts_tablet_manager.cc:1918] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250626 19:57:37.108532 4391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 70) took 1782 ms (client timeout 1000 ms). Trace:
I20250626 19:57:37.110498 5353 tablet_bootstrap.cc:492] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:37.110996 5353 ts_tablet_manager.cc:1397] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.122s user 0.009s sys 0.002s
I20250626 19:57:37.064090 5366 raft_consensus.cc:383] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:37.112080 5595 raft_consensus.cc:491] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:37.107949 5438 log.cc:1199] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/499f6ae3262940b4859c14a53722482f
I20250626 19:57:37.113783 5353 raft_consensus.cc:357] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.114391 5353 raft_consensus.cc:383] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:37.112335 5366 raft_consensus.cc:738] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:37.114641 5353 raft_consensus.cc:738] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:37.116036 5353 consensus_queue.cc:260] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.115657 5366 consensus_queue.cc:260] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.112617 5595 raft_consensus.cc:513] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:37.121582 5353 ts_tablet_manager.cc:1428] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.010s user 0.004s sys 0.001s
I20250626 19:57:37.122467 5433 ts_tablet_manager.cc:1918] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:37.122845 5433 log.cc:1199] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/fd6543699e254400b3516aece4486fe0
I20250626 19:57:37.123589 5433 ts_tablet_manager.cc:1939] T fd6543699e254400b3516aece4486fe0 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:37.124390 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 44) took 2586 ms. Trace:
I20250626 19:57:37.124341 5438 ts_tablet_manager.cc:1939] T 499f6ae3262940b4859c14a53722482f P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:37.127236 5366 ts_tablet_manager.cc:1428] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.081s user 0.003s sys 0.003s
I20250626 19:57:37.144428 5569 raft_consensus.cc:491] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:37.144850 5569 raft_consensus.cc:513] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.147363 5569 leader_election.cc:290] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:37.149864 4950 tablet_service.cc:1468] Processing CreateTablet for tablet f6ec952321924098a142f1438a283aab (DEFAULT_TABLE table=default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:37.151006 4950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6ec952321924098a142f1438a283aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:37.151863 5595 leader_election.cc:290] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:37.152621 4396 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 36997dab0ab3430bba63a90805759b30
I20250626 19:57:37.153962 4396 catalog_manager.cc:6652] ReplaceTablet: tablet 36997dab0ab3430bba63a90805759b30 deleted and replaced by tablet fafbf98e7f4544b6ad96c1c5ab6c82bc
I20250626 19:57:37.154594 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c03691d021e4ee19d5bb287ab633a3c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:37.155261 4653 tablet_service.cc:1468] Processing CreateTablet for tablet f6ec952321924098a142f1438a283aab (DEFAULT_TABLE table=default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:37.155957 4396 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_3cb641007af847a68d8f13f80829fe0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:37.156734 5410 tablet_bootstrap.cc:654] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:37.157457 4396 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cb641007af847a68d8f13f80829fe0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:37.157431 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet 36997dab0ab3430bba63a90805759b30 with delete_type TABLET_DATA_DELETED (replaced by tablet fafbf98e7f4544b6ad96c1c5ab6c82bc) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:37.124743 5433 rpcz_store.cc:276] 0626 19:57:34.538272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:34.538711 (+ 439us) service_pool.cc:225] Handling call
0626 19:57:36.319785 (+1781074us) ts_tablet_manager.cc:1145] Deleting tablet fd6543699e254400b3516aece4486fe0
0626 19:57:36.319826 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:36.902229 (+582403us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.122233 (+220004us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.124338 (+ 2105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":131840,"tablet-delete.queue_time_us":1776189}
I20250626 19:57:37.167379 5433 tablet_replica.cc:331] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:37.170981 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:37.173089 5353 tablet_bootstrap.cc:492] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:37.180781 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet fd6543699e254400b3516aece4486fe0 (table default.Table_fde5b6bd4ee8417e85ffccad3bd2a581 [id=ab0bf254569748ba8d31931d4bfb105d]) successfully deleted
I20250626 19:57:37.180567 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet 36997dab0ab3430bba63a90805759b30 with delete_type TABLET_DATA_DELETED (replaced by tablet fafbf98e7f4544b6ad96c1c5ab6c82bc) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:37.179531 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c0338c05a974e0f96865398722c5e8f" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:37.184834 4385 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4c0338c05a974e0f96865398722c5e8f consensus state has no leader
W20250626 19:57:37.138993 4391 rpcz_store.cc:269] 0626 19:57:35.326660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:35.339753 (+ 13093us) service_pool.cc:225] Handling call
0626 19:57:36.278850 (+939097us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:36.278879 (+ 29us) write_op.cc:454] Released schema lock
0626 19:57:37.108466 (+829587us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:36.224639 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:36.225015 (+ 376us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:36.225037 (+ 22us) write_op.cc:435] Acquired schema lock
0626 19:57:36.225050 (+ 13us) tablet.cc:660] Decoding operations
0626 19:57:36.225450 (+ 400us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:36.225504 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:36.225530 (+ 26us) tablet.cc:683] Acquiring locks for 3 operations
0626 19:57:36.226325 (+ 795us) tablet.cc:699] Row locks acquired
0626 19:57:36.226339 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:57:36.226448 (+ 109us) write_op.cc:270] Start()
0626 19:57:36.226541 (+ 93us) write_op.cc:276] Timestamp: P: 1750967856226530 usec, L: 0
0626 19:57:36.226557 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:57:36.227113 (+ 556us) log.cc:844] Serialized 1284 byte log entry
0626 19:57:36.236439 (+ 9326us) op_driver.cc:464] REPLICATION: finished
0626 19:57:36.236843 (+ 404us) write_op.cc:301] APPLY: starting
0626 19:57:36.236898 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:36.237022 (+ 124us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:36.237130 (+ 108us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:36.237431 (+ 301us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:36.237611 (+ 180us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0626 19:57:36.237652 (+ 41us) write_op.cc:312] APPLY: finished
0626 19:57:36.237878 (+ 226us) log.cc:844] Serialized 35 byte log entry
0626 19:57:36.238020 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:36.238110 (+ 90us) write_op.cc:454] Released schema lock
0626 19:57:36.238140 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":2321,"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":246,"apply.run_cpu_time_us":1502,"apply.run_wall_time_us":1501,"num_ops":3,"prepare.queue_time_us":110,"prepare.run_cpu_time_us":2803,"prepare.run_wall_time_us":4873,"raft.queue_time_us":961,"raft.run_cpu_time_us":658,"raft.run_wall_time_us":659,"replication_time_us":9782}]]}
I20250626 19:57:37.187384 4390 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet cb34ffcd4da04e32a1220b3f298ec22d
I20250626 19:57:37.193759 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:37.195608 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:37.217142 5431 ts_tablet_manager.cc:1918] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:37.218713 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_6b965a2deba74eeeb9601ca4f736d338"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:37.221415 4396 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_fbcc45f0e38f4ed184900c0f58067165"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:37.222946 4396 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbcc45f0e38f4ed184900c0f58067165 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:37.224124 5433 raft_consensus.cc:2241] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:37.224668 5433 raft_consensus.cc:2270] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:37.226466 5433 ts_tablet_manager.cc:1905] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:37.227543 5353 tablet_bootstrap.cc:654] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:37.232801 5431 log.cc:1199] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/2ab0320af47049cca3ca87172b9fe5e5
I20250626 19:57:37.239351 5431 ts_tablet_manager.cc:1939] T 2ab0320af47049cca3ca87172b9fe5e5 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
W20250626 19:57:37.222939 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b965a2deba74eeeb9601ca4f736d338 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:37.247529 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6ec952321924098a142f1438a283aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:37.256439 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 2ab0320af47049cca3ca87172b9fe5e5 (table default.Table_d23ee99af7b845b78d2200b962f681a6 [id=fc725d6df6774d4e922e2d98d461b111]) successfully deleted
W20250626 19:57:37.265424 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 36997dab0ab3430bba63a90805759b30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36997dab0ab3430bba63a90805759b30 already in progress: creating tablet
W20250626 19:57:37.267514 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet cb34ffcd4da04e32a1220b3f298ec22d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb34ffcd4da04e32a1220b3f298ec22d already in progress: creating tablet
I20250626 19:57:37.148211 4801 tablet_service.cc:1468] Processing CreateTablet for tablet f6ec952321924098a142f1438a283aab (DEFAULT_TABLE table=default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:37.277716 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6ec952321924098a142f1438a283aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:37.149295 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c0338c05a974e0f96865398722c5e8f" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:37.149296 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 36) took 3087 ms. Trace:
I20250626 19:57:37.280292 5438 rpcz_store.cc:276] 0626 19:57:34.062380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:34.074898 (+ 12518us) service_pool.cc:225] Handling call
0626 19:57:37.019645 (+2944747us) ts_tablet_manager.cc:1145] Deleting tablet 499f6ae3262940b4859c14a53722482f
0626 19:57:37.019700 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:37.036275 (+ 16575us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.101769 (+ 65494us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.149242 (+ 47473us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2944067}
I20250626 19:57:37.281446 5438 tablet_replica.cc:331] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:37.282151 5438 raft_consensus.cc:2241] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:37.282573 5438 raft_consensus.cc:2270] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:37.284142 5438 ts_tablet_manager.cc:1905] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:37.159961 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c03691d021e4ee19d5bb287ab633a3c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:37.160738 4796 tablet_service.cc:1515] Processing DeleteTablet for tablet 36997dab0ab3430bba63a90805759b30 with delete_type TABLET_DATA_DELETED (replaced by tablet fafbf98e7f4544b6ad96c1c5ab6c82bc) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:37.165208 5429 raft_consensus.cc:491] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:37.286294 4817 raft_consensus.cc:2391] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:37.286164 5429 raft_consensus.cc:513] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.287966 5429 leader_election.cc:290] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:37.288021 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:37.289134 4609 leader_election.cc:304] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:37.290118 5595 raft_consensus.cc:3058] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:37.198693 4756 consensus_peers.cc:487] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b3bed19e7e34d24b995840a3bde5ef0. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:37.210160 4800 tablet_service.cc:1515] Processing DeleteTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:40742
W20250626 19:57:37.295121 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 96628us
I20250626 19:57:37.297331 4911 leader_election.cc:304] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:37.298275 5569 raft_consensus.cc:2747] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:37.299561 5561 raft_consensus.cc:491] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:37.299800 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65af5ae255d74143976392bccc50f3ba" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:37.299962 5561 raft_consensus.cc:513] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.300385 4973 raft_consensus.cc:2466] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:37.301676 5561 leader_election.cc:290] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:37.302078 5561 raft_consensus.cc:491] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:37.303225 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:37.305281 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 5e20ea7c17c34fbd9c117275e3c14466 (table default.Table_c0263194234444158f569a2abbfa91b5 [id=089059dab32a41179048a6eb4473a99a]). Expected version 2 got 0
W20250626 19:57:37.305773 4756 consensus_peers.cc:487] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:37.306399 4756 leader_election.cc:304] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:37.309067 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fb3692f4ddb4ff89c1817da4a5cb927" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:37.309587 5506 raft_consensus.cc:2802] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:37.309908 5506 raft_consensus.cc:491] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:37.310215 5506 raft_consensus.cc:3058] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:37.311976 5438 ts_tablet_manager.cc:1918] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:37.312309 5438 log.cc:1199] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/fd6543699e254400b3516aece4486fe0
I20250626 19:57:37.312906 5438 ts_tablet_manager.cc:1939] T fd6543699e254400b3516aece4486fe0 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:37.315248 5433 ts_tablet_manager.cc:1918] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:37.315663 5433 log.cc:1199] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/4b3bed19e7e34d24b995840a3bde5ef0
I20250626 19:57:37.316394 5433 ts_tablet_manager.cc:1939] T 4b3bed19e7e34d24b995840a3bde5ef0 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:37.319310 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 45) took 2461 ms. Trace:
I20250626 19:57:37.319859 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65af5ae255d74143976392bccc50f3ba" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:37.314733 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 40) took 2775 ms. Trace:
I20250626 19:57:37.320174 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fb3692f4ddb4ff89c1817da4a5cb927" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:37.320634 5438 rpcz_store.cc:276] 0626 19:57:34.539487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:34.539738 (+ 251us) service_pool.cc:225] Handling call
0626 19:57:37.281296 (+2741558us) ts_tablet_manager.cc:1145] Deleting tablet fd6543699e254400b3516aece4486fe0
0626 19:57:37.281342 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:37.300129 (+ 18787us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.311760 (+ 11631us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.314670 (+ 2910us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2741049}
I20250626 19:57:37.321806 5438 tablet_replica.cc:331] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
W20250626 19:57:37.321494 4755 leader_election.cc:343] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:37.322443 4755 leader_election.cc:343] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:37.322638 5438 raft_consensus.cc:2241] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:37.302421 5561 raft_consensus.cc:513] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.323758 5438 raft_consensus.cc:2270] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:37.324515 5561 leader_election.cc:290] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:37.325531 5438 ts_tablet_manager.cc:1905] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:37.331617 4756 consensus_peers.cc:487] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ab0320af47049cca3ca87172b9fe5e5. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:37.331774 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94cefa59de044a36a2afaaf5531128d8" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:37.332154 4756 leader_election.cc:343] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:37.332778 4756 leader_election.cc:304] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:37.308320 4399 catalog_manager.cc:5582] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:37.334327 5429 raft_consensus.cc:2747] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:37.335417 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 3c03691d021e4ee19d5bb287ab633a3c (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:37.337056 5595 raft_consensus.cc:2747] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:37.339257 5438 ts_tablet_manager.cc:1918] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:37.339610 5438 log.cc:1199] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/4b3bed19e7e34d24b995840a3bde5ef0
I20250626 19:57:37.340428 5438 ts_tablet_manager.cc:1939] T 4b3bed19e7e34d24b995840a3bde5ef0 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:37.340281 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94cefa59de044a36a2afaaf5531128d8" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:37.319641 5433 rpcz_store.cc:276] 0626 19:57:34.857540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:34.857647 (+ 107us) service_pool.cc:225] Handling call
0626 19:57:37.167220 (+2309573us) ts_tablet_manager.cc:1145] Deleting tablet 4b3bed19e7e34d24b995840a3bde5ef0
0626 19:57:37.167267 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:37.278097 (+110830us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.315014 (+ 36917us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.319262 (+ 4248us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41493,"tablet-delete.queue_time_us":2308899}
I20250626 19:57:37.342079 5433 tablet_replica.cc:331] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:37.341910 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 4b3bed19e7e34d24b995840a3bde5ef0 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]) successfully deleted
I20250626 19:57:37.342867 5433 raft_consensus.cc:2241] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:37.343456 5433 raft_consensus.cc:2270] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:37.345172 5433 ts_tablet_manager.cc:1905] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:37.347457 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:37.351428 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 499f6ae3262940b4859c14a53722482f (table default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]) successfully deleted
W20250626 19:57:37.333753 4756 leader_election.cc:343] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:37.359428 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet fd6543699e254400b3516aece4486fe0 (table default.Table_fde5b6bd4ee8417e85ffccad3bd2a581 [id=ab0bf254569748ba8d31931d4bfb105d]) successfully deleted
I20250626 19:57:37.359656 5433 ts_tablet_manager.cc:1918] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:57:37.360122 5433 log.cc:1199] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/1cb6bda529624ac5b5a87a03b4a02d56
I20250626 19:57:37.360540 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:37.360945 5433 ts_tablet_manager.cc:1939] T 1cb6bda529624ac5b5a87a03b4a02d56 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:37.363042 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 51) took 2270 ms. Trace:
I20250626 19:57:37.363394 5433 rpcz_store.cc:276] 0626 19:57:35.092790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:35.107700 (+ 14910us) service_pool.cc:225] Handling call
0626 19:57:37.341920 (+2234220us) ts_tablet_manager.cc:1145] Deleting tablet 1cb6bda529624ac5b5a87a03b4a02d56
0626 19:57:37.341964 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:37.352009 (+ 10045us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.359459 (+ 7450us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.362997 (+ 3538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2143803}
I20250626 19:57:37.364542 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 1cb6bda529624ac5b5a87a03b4a02d56 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]) successfully deleted
I20250626 19:57:37.371179 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 41) took 2484 ms. Trace:
I20250626 19:57:37.371500 5438 rpcz_store.cc:276] 0626 19:57:34.886999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:34.887093 (+ 94us) service_pool.cc:225] Handling call
0626 19:57:37.321656 (+2434563us) ts_tablet_manager.cc:1145] Deleting tablet 4b3bed19e7e34d24b995840a3bde5ef0
0626 19:57:37.321694 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:37.332685 (+ 10991us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.339028 (+ 6343us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.371117 (+ 32089us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2434091}
I20250626 19:57:37.372494 5438 tablet_replica.cc:331] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:37.373175 5438 raft_consensus.cc:2241] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:37.373633 5438 raft_consensus.cc:2270] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:37.375062 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 4b3bed19e7e34d24b995840a3bde5ef0 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]) successfully deleted
I20250626 19:57:37.376222 5438 ts_tablet_manager.cc:1905] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:37.379871 5506 raft_consensus.cc:513] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.381693 5506 leader_election.cc:290] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
W20250626 19:57:37.383637 4755 leader_election.cc:343] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:37.383860 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65af5ae255d74143976392bccc50f3ba" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:37.384387 4978 raft_consensus.cc:3058] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:37.385912 4755 leader_election.cc:304] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:37.391587 4978 raft_consensus.cc:2466] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:37.392210 5433 tablet_replica.cc:331] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:37.393247 5433 raft_consensus.cc:2241] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:37.393956 4756 leader_election.cc:304] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:37.396317 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65af5ae255d74143976392bccc50f3ba" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:37.399412 5433 raft_consensus.cc:2270] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:37.401083 5433 ts_tablet_manager.cc:1905] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:37.394173 5561 raft_consensus.cc:2747] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:37.394542 5429 consensus_queue.cc:1035] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:57:37.404311 4755 leader_election.cc:343] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:37.405365 5429 raft_consensus.cc:2802] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:37.405957 5429 raft_consensus.cc:695] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:37.406705 5429 consensus_queue.cc:237] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.412170 5353 tablet_bootstrap.cc:492] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:37.412591 5353 ts_tablet_manager.cc:1397] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.240s user 0.007s sys 0.004s
I20250626 19:57:37.412873 4396 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_c580e8a6ee5d417caa2f7d6811a68349" } modify_external_catalogs: true
I20250626 19:57:37.413372 4396 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_c580e8a6ee5d417caa2f7d6811a68349" } modify_external_catalogs: true
I20250626 19:57:37.401572 5366 tablet_bootstrap.cc:492] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:37.415086 5353 raft_consensus.cc:357] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.415861 5353 raft_consensus.cc:383] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:37.416179 5353 raft_consensus.cc:738] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:37.416934 5353 consensus_queue.cc:260] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.424060 5353 ts_tablet_manager.cc:1428] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250626 19:57:37.425432 4390 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_7a151cd2d08a4ec9aca048d41f8a5c04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:37.425678 4392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_f85ad493ebbf4e3b8c7eea1edfcf6a67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:37.427332 4390 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a151cd2d08a4ec9aca048d41f8a5c04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:57:37.427393 4392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f85ad493ebbf4e3b8c7eea1edfcf6a67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:37.435738 5569 raft_consensus.cc:491] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:37.436199 5569 raft_consensus.cc:513] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.438050 5569 leader_election.cc:290] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:37.440521 5353 tablet_bootstrap.cc:492] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:37.462248 5353 tablet_bootstrap.cc:654] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:37.465210 4383 consensus_peers.cc:487] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af -> Peer e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823): Couldn't send request to peer e6c58006b6be41c1bd6bdb5eb010d360. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250626 19:57:37.466032 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43aed8f07844c5ca727ae6df63896df" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:37.470587 5429 raft_consensus.cc:491] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:37.471158 5429 raft_consensus.cc:513] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.480605 5429 leader_election.cc:290] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:37.472540 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:37.488685 5366 tablet_bootstrap.cc:654] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:37.483739 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e60b4d53c40c4dffab928f0e3fb4a80b" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:37.495949 4756 leader_election.cc:343] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:37.474619 4393 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:37.509893 5353 tablet_bootstrap.cc:492] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:37.510325 5353 ts_tablet_manager.cc:1397] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.004s
I20250626 19:57:37.508044 4393 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:37.512694 5353 raft_consensus.cc:357] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.513304 5353 raft_consensus.cc:383] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:37.513554 5353 raft_consensus.cc:738] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:37.510437 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43aed8f07844c5ca727ae6df63896df" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:37.539448 5353 consensus_queue.cc:260] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.543205 4393 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5e20ea7c17c34fbd9c117275e3c14466 (table default.Table_c0263194234444158f569a2abbfa91b5 [id=089059dab32a41179048a6eb4473a99a]). Expected version 2 got 0
I20250626 19:57:37.556524 5353 ts_tablet_manager.cc:1428] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.046s user 0.005s sys 0.000s
I20250626 19:57:37.563001 5410 tablet_bootstrap.cc:492] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:37.563519 5410 ts_tablet_manager.cc:1397] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.505s user 0.031s sys 0.015s
I20250626 19:57:37.565481 5438 ts_tablet_manager.cc:1918] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:57:37.565883 5438 log.cc:1199] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/1cb6bda529624ac5b5a87a03b4a02d56
I20250626 19:57:37.565639 5410 raft_consensus.cc:357] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.566385 5410 raft_consensus.cc:383] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:37.566740 5410 raft_consensus.cc:738] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:37.567493 5410 consensus_queue.cc:260] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.569355 5438 ts_tablet_manager.cc:1939] T 1cb6bda529624ac5b5a87a03b4a02d56 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:37.570703 5353 tablet_bootstrap.cc:492] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:37.569375 5366 tablet_bootstrap.cc:492] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:37.575467 5366 ts_tablet_manager.cc:1397] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.174s user 0.010s sys 0.000s
I20250626 19:57:37.576391 5410 ts_tablet_manager.cc:1428] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20250626 19:57:37.577059 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 47) took 2461 ms. Trace:
I20250626 19:57:37.577359 5438 rpcz_store.cc:276] 0626 19:57:35.115921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:35.138219 (+ 22298us) service_pool.cc:225] Handling call
0626 19:57:37.372353 (+2234134us) ts_tablet_manager.cc:1145] Deleting tablet 1cb6bda529624ac5b5a87a03b4a02d56
0626 19:57:37.372392 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:37.435954 (+ 63562us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.565237 (+129283us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.577003 (+ 11766us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2233496}
I20250626 19:57:37.565179 4392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_e298729c4c744ce3834f9f02a49d4349"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:37.578438 5438 tablet_replica.cc:331] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:37.578455 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 1cb6bda529624ac5b5a87a03b4a02d56 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]) successfully deleted
W20250626 19:57:37.579977 4392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e298729c4c744ce3834f9f02a49d4349 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:37.581543 5366 raft_consensus.cc:357] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.581537 5410 tablet_bootstrap.cc:492] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:37.582188 5366 raft_consensus.cc:383] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:37.586827 5366 raft_consensus.cc:738] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:37.587525 5366 consensus_queue.cc:260] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.595113 5438 raft_consensus.cc:2241] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:37.595105 4393 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:37.595570 5438 raft_consensus.cc:2270] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:37.579818 5433 ts_tablet_manager.cc:1918] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:37.603436 5433 log.cc:1199] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/c92deaa237064dee8f866e3a7a355ca4
I20250626 19:57:37.604251 5438 ts_tablet_manager.cc:1905] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:37.607728 5353 tablet_bootstrap.cc:654] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:37.608630 5433 ts_tablet_manager.cc:1939] T c92deaa237064dee8f866e3a7a355ca4 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:37.608644 4979 raft_consensus.cc:3058] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:37.609375 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 66) took 1242 ms. Trace:
I20250626 19:57:37.609689 5433 rpcz_store.cc:276] 0626 19:57:36.366943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.371141 (+ 4198us) service_pool.cc:225] Handling call
0626 19:57:37.364639 (+993498us) ts_tablet_manager.cc:1145] Deleting tablet c92deaa237064dee8f866e3a7a355ca4
0626 19:57:37.392042 (+ 27403us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:37.485985 (+ 93943us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.579605 (+ 93620us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.609320 (+ 29715us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":985996}
I20250626 19:57:37.612303 4392 master_service.cc:948] ReplaceTablet: received request to replace tablet 7391b9eaea3e426ba4118fc8b53b72a6 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:37.613116 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 67) took 1241 ms. Trace:
I20250626 19:57:37.613413 5433 rpcz_store.cc:276] 0626 19:57:36.371811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.374875 (+ 3064us) service_pool.cc:225] Handling call
0626 19:57:37.610614 (+1235739us) ts_tablet_manager.cc:1145] Deleting tablet 82823fd9ceb1485386616319fa7aeb43
0626 19:57:37.610655 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:37.613060 (+ 2405us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1231617}
I20250626 19:57:37.614343 5433 tablet_replica.cc:331] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:37.615129 5433 raft_consensus.cc:2241] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:37.616006 5433 raft_consensus.cc:2270] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:37.617789 5433 ts_tablet_manager.cc:1905] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:37.620831 4393 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 3c03691d021e4ee19d5bb287ab633a3c (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:37.626565 5366 ts_tablet_manager.cc:1428] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.051s user 0.005s sys 0.000s
I20250626 19:57:37.630489 5353 tablet_bootstrap.cc:492] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:37.631042 5353 ts_tablet_manager.cc:1397] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.002s
I20250626 19:57:37.633167 5353 raft_consensus.cc:357] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.633721 5353 raft_consensus.cc:383] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:37.633972 5353 raft_consensus.cc:738] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:37.635816 5366 tablet_bootstrap.cc:492] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:37.648061 5353 consensus_queue.cc:260] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.650023 5353 ts_tablet_manager.cc:1428] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.019s user 0.002s sys 0.001s
I20250626 19:57:37.650826 5353 tablet_bootstrap.cc:492] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:37.653903 4392 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 7391b9eaea3e426ba4118fc8b53b72a6
I20250626 19:57:37.655998 4392 catalog_manager.cc:6652] ReplaceTablet: tablet 7391b9eaea3e426ba4118fc8b53b72a6 deleted and replaced by tablet d8399c219d0e4485bb7926eefd78f210
I20250626 19:57:37.656025 4950 tablet_service.cc:1515] Processing DeleteTablet for tablet 7391b9eaea3e426ba4118fc8b53b72a6 with delete_type TABLET_DATA_DELETED (replaced by tablet d8399c219d0e4485bb7926eefd78f210) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:37.656668 5431 tablet_replica.cc:331] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:37.656757 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 7391b9eaea3e426ba4118fc8b53b72a6 with delete_type TABLET_DATA_DELETED (replaced by tablet d8399c219d0e4485bb7926eefd78f210) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:37.657406 5431 raft_consensus.cc:2241] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:37.657866 5431 raft_consensus.cc:2270] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:37.662257 5431 ts_tablet_manager.cc:1905] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:37.678821 5366 tablet_bootstrap.cc:654] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:37.684684 4395 catalog_manager.cc:5582] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:37.693404 5438 ts_tablet_manager.cc:1918] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:37.693943 5438 log.cc:1199] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/c92deaa237064dee8f866e3a7a355ca4
I20250626 19:57:37.694657 5438 ts_tablet_manager.cc:1939] T c92deaa237064dee8f866e3a7a355ca4 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:37.699841 4392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_32a604ea7e0d4d9a8b477c5e6c1e87d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:37.701416 4392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32a604ea7e0d4d9a8b477c5e6c1e87d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:37.701959 5615 consensus_queue.cc:1035] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:37.703991 5431 ts_tablet_manager.cc:1918] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:37.704322 5431 log.cc:1199] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/7391b9eaea3e426ba4118fc8b53b72a6
I20250626 19:57:37.704978 5431 ts_tablet_manager.cc:1939] T 7391b9eaea3e426ba4118fc8b53b72a6 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:37.705474 5353 tablet_bootstrap.cc:654] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:37.708714 5615 consensus_queue.cc:1035] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:37.713677 5429 raft_consensus.cc:491] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:37.714131 5429 raft_consensus.cc:513] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:37.716229 5429 leader_election.cc:290] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:37.717814 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c990e4eca5cf4a8e8295e9599cf5be29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:37.718381 4979 raft_consensus.cc:2466] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:37.718907 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 60) took 1668 ms. Trace:
I20250626 19:57:37.719744 4756 leader_election.cc:304] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:37.720369 5429 raft_consensus.cc:2802] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 19:57:37.720758 5429 raft_consensus.cc:491] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:37.721154 5429 raft_consensus.cc:3058] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:57:37.719242 5438 rpcz_store.cc:276] 0626 19:57:36.050428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.050528 (+ 100us) service_pool.cc:225] Handling call
0626 19:57:37.578297 (+1527769us) ts_tablet_manager.cc:1145] Deleting tablet c92deaa237064dee8f866e3a7a355ca4
0626 19:57:37.578335 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:37.640473 (+ 62138us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.693118 (+ 52645us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:37.718841 (+ 25723us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1527231}
I20250626 19:57:37.723763 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 62) took 1565 ms. Trace:
I20250626 19:57:37.723817 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet c92deaa237064dee8f866e3a7a355ca4 (table default.Table_a2a4a2c9c59248dfb73f7db218b0d982 [id=0d00dcf2b9f64037b166b83f38e1a72a]) successfully deleted
I20250626 19:57:37.724056 5438 rpcz_store.cc:276] 0626 19:57:36.157990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.158096 (+ 106us) service_pool.cc:225] Handling call
0626 19:57:37.723536 (+1565440us) ts_tablet_manager.cc:1145] Deleting tablet afd1fafb53434957b002fad5c77431d4
0626 19:57:37.723578 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:37.723716 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1564934}
I20250626 19:57:37.725054 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 63) took 1566 ms. Trace:
W20250626 19:57:37.725416 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet afd1fafb53434957b002fad5c77431d4 because the tablet was not found. No further retry: Not found: Tablet not found: afd1fafb53434957b002fad5c77431d4
I20250626 19:57:37.725333 5438 rpcz_store.cc:276] 0626 19:57:36.158220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.158435 (+ 215us) service_pool.cc:225] Handling call
0626 19:57:37.724836 (+1566401us) ts_tablet_manager.cc:1145] Deleting tablet 512dc911bfce49b29c6fc8cd90523723
0626 19:57:37.724875 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:37.725014 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1565876}
W20250626 19:57:37.726470 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 512dc911bfce49b29c6fc8cd90523723 because the tablet was not found. No further retry: Not found: Tablet not found: 512dc911bfce49b29c6fc8cd90523723
I20250626 19:57:37.726897 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 7391b9eaea3e426ba4118fc8b53b72a6 (table default.Table_e5f9396b41eb4c5eb98ec42ec48d7cdc [id=bad1d85ef3a347ab9abe78289306ca4d]) successfully deleted
I20250626 19:57:37.746066 5615 raft_consensus.cc:491] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:37.746716 5615 raft_consensus.cc:513] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.750602 5353 tablet_bootstrap.cc:492] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:37.751122 5353 ts_tablet_manager.cc:1397] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.100s user 0.011s sys 0.000s
I20250626 19:57:37.753881 5353 raft_consensus.cc:357] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.755232 5615 leader_election.cc:290] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:37.767483 4392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_73a1fc5db8b944b3bddfc9cc2d03b002"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:37.769204 4392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73a1fc5db8b944b3bddfc9cc2d03b002 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:37.726207 5438 tablet_replica.cc:331] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:37.775976 5438 raft_consensus.cc:2241] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:37.776168 5366 tablet_bootstrap.cc:492] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:37.776523 5366 ts_tablet_manager.cc:1397] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.141s user 0.014s sys 0.013s
I20250626 19:57:37.776472 5438 raft_consensus.cc:2270] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:37.778344 5438 ts_tablet_manager.cc:1905] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:37.788203 4392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_e4d8a4cf39764c5fa002644d60f964b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:37.790074 4392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4d8a4cf39764c5fa002644d60f964b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:37.754679 5353 raft_consensus.cc:383] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:37.791795 5353 raft_consensus.cc:738] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:37.687247 4395 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:37.793926 4395 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:37.796811 5353 consensus_queue.cc:260] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.815601 4392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_6dc13f5ceb674e40919de2ca36d9427a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:37.817294 4392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6dc13f5ceb674e40919de2ca36d9427a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:37.824457 5353 ts_tablet_manager.cc:1428] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.073s user 0.007s sys 0.000s
I20250626 19:57:37.825374 5353 tablet_bootstrap.cc:492] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:37.828516 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e60b4d53c40c4dffab928f0e3fb4a80b" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:37.829128 4821 raft_consensus.cc:2466] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:37.835397 4911 leader_election.cc:304] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:37.836267 5615 raft_consensus.cc:2802] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:37.836620 5615 raft_consensus.cc:491] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:37.836994 5615 raft_consensus.cc:3058] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:37.848765 5429 raft_consensus.cc:513] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:37.850157 5353 tablet_bootstrap.cc:654] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:37.851210 5429 leader_election.cc:290] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 2 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:37.860376 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c990e4eca5cf4a8e8295e9599cf5be29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:37.860971 4979 raft_consensus.cc:3058] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:57:37.797328 4395 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 5e20ea7c17c34fbd9c117275e3c14466 (table default.Table_c0263194234444158f569a2abbfa91b5 [id=089059dab32a41179048a6eb4473a99a]). Expected version 2 got 0
I20250626 19:57:37.868026 4395 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:37.869794 4395 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 3c03691d021e4ee19d5bb287ab633a3c (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:37.862542 5366 raft_consensus.cc:357] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.871340 5366 raft_consensus.cc:383] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:37.872615 4979 raft_consensus.cc:2466] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 2.
I20250626 19:57:37.873116 5366 raft_consensus.cc:738] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:37.846683 5410 tablet_bootstrap.cc:654] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:37.875162 5366 consensus_queue.cc:260] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.877228 5366 ts_tablet_manager.cc:1428] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.100s user 0.005s sys 0.001s
I20250626 19:57:37.878084 5366 tablet_bootstrap.cc:492] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:37.890059 5615 raft_consensus.cc:513] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.889598 4756 leader_election.cc:304] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:37.891283 5429 raft_consensus.cc:2802] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 19:57:37.892294 5429 raft_consensus.cc:695] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 2 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:37.893095 5429 consensus_queue.cc:237] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } }
I20250626 19:57:37.901711 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet c92deaa237064dee8f866e3a7a355ca4 (table default.Table_a2a4a2c9c59248dfb73f7db218b0d982 [id=0d00dcf2b9f64037b166b83f38e1a72a]) successfully deleted
I20250626 19:57:37.905601 5624 raft_consensus.cc:491] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:37.906210 5624 raft_consensus.cc:513] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.908512 5624 leader_election.cc:290] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:37.912385 5366 tablet_bootstrap.cc:654] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:37.925997 4392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_2e7897256c7140aeb545751922965c14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:37.927507 4392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e7897256c7140aeb545751922965c14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:37.933210 5366 tablet_bootstrap.cc:492] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:37.933614 5366 ts_tablet_manager.cc:1397] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.007s
I20250626 19:57:37.935237 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e60b4d53c40c4dffab928f0e3fb4a80b" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:37.937955 5366 raft_consensus.cc:357] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.938583 5366 raft_consensus.cc:383] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:37.940903 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c990e4eca5cf4a8e8295e9599cf5be29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:37.941556 4669 raft_consensus.cc:2466] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:37.938845 5366 raft_consensus.cc:738] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:37.943629 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e60b4d53c40c4dffab928f0e3fb4a80b" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:37.944953 4674 raft_consensus.cc:3058] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:37.946205 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 82823fd9ceb1485386616319fa7aeb43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82823fd9ceb1485386616319fa7aeb43 already in progress: creating tablet
I20250626 19:57:37.948716 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet 7391b9eaea3e426ba4118fc8b53b72a6 with delete_type TABLET_DATA_DELETED (replaced by tablet d8399c219d0e4485bb7926eefd78f210) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:37.951040 4978 raft_consensus.cc:3058] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:37.953610 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:37.951059 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61fcd55f6d7b4a369383d4f73d5f483d" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:37.942646 5366 consensus_queue.cc:260] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:37.971467 5366 ts_tablet_manager.cc:1428] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20250626 19:57:37.974129 5366 tablet_bootstrap.cc:492] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:37.974840 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c990e4eca5cf4a8e8295e9599cf5be29" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:37.975436 4674 raft_consensus.cc:3058] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:57:37.984846 5366 tablet_bootstrap.cc:654] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:37.987200 5615 leader_election.cc:290] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:37.999193 4755 leader_election.cc:343] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:37.999716 4755 leader_election.cc:304] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:38.009168 5438 ts_tablet_manager.cc:1918] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:38.013290 5615 raft_consensus.cc:491] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:38.013742 5615 raft_consensus.cc:513] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.013900 5429 raft_consensus.cc:2747] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:38.016346 5438 log.cc:1199] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/a31f1a7739544706bff2a500d6026086
I20250626 19:57:38.017844 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e60b4d53c40c4dffab928f0e3fb4a80b" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:38.020157 5353 tablet_bootstrap.cc:492] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:38.020563 5353 ts_tablet_manager.cc:1397] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.195s user 0.009s sys 0.000s
I20250626 19:57:38.018591 4819 raft_consensus.cc:3058] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.024585 4392 master_service.cc:948] ReplaceTablet: received request to replace tablet 3c03691d021e4ee19d5bb287ab633a3c from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:38.018544 5438 ts_tablet_manager.cc:1939] T a31f1a7739544706bff2a500d6026086 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:38.028294 5615 leader_election.cc:290] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:38.022699 5353 raft_consensus.cc:357] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.034042 5353 raft_consensus.cc:383] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.035195 4950 tablet_service.cc:1468] Processing CreateTablet for tablet 69bd106066b6492bb7ea03d15a18df00 (DEFAULT_TABLE table=default.Table_3e24f21638574c22b00dd906bb373acb [id=65b3f2207439437190186ccbc652f43a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.036306 4950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69bd106066b6492bb7ea03d15a18df00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.034301 5353 raft_consensus.cc:738] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:38.042097 4954 tablet_service.cc:1468] Processing CreateTablet for tablet 3143181fae16411888d74f9152f3d5cc (DEFAULT_TABLE table=default.Table_ad7eb0792b7444b2891bf198ef117218 [id=8ae02648a2e946c18cbb5f2d52ce375a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.043042 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3143181fae16411888d74f9152f3d5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.040166 5353 consensus_queue.cc:260] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.044117 5353 ts_tablet_manager.cc:1428] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20250626 19:57:38.044906 5353 tablet_bootstrap.cc:492] T 419c4018256546ef9431ac2db619ee33 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:38.038975 5433 ts_tablet_manager.cc:1918] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:38.045717 5433 log.cc:1199] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/a31f1a7739544706bff2a500d6026086
I20250626 19:57:38.046578 5433 ts_tablet_manager.cc:1939] T a31f1a7739544706bff2a500d6026086 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:38.049979 4958 tablet_service.cc:1468] Processing CreateTablet for tablet 781f8ee8c8ad49c6b2391fcdf20bdfdb (DEFAULT_TABLE table=default.Table_c62b1259fcee4798bbcc9a86437cf004 [id=eb61abd19bed46288250e4a8f02c39c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.050963 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781f8ee8c8ad49c6b2391fcdf20bdfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.060149 4653 tablet_service.cc:1468] Processing CreateTablet for tablet 69bd106066b6492bb7ea03d15a18df00 (DEFAULT_TABLE table=default.Table_3e24f21638574c22b00dd906bb373acb [id=65b3f2207439437190186ccbc652f43a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.060865 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 69bd106066b6492bb7ea03d15a18df00 (DEFAULT_TABLE table=default.Table_3e24f21638574c22b00dd906bb373acb [id=65b3f2207439437190186ccbc652f43a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.062367 4652 tablet_service.cc:1468] Processing CreateTablet for tablet 3143181fae16411888d74f9152f3d5cc (DEFAULT_TABLE table=default.Table_ad7eb0792b7444b2891bf198ef117218 [id=8ae02648a2e946c18cbb5f2d52ce375a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.063573 5353 tablet_bootstrap.cc:654] T 419c4018256546ef9431ac2db619ee33 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:38.063576 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 3143181fae16411888d74f9152f3d5cc (DEFAULT_TABLE table=default.Table_ad7eb0792b7444b2891bf198ef117218 [id=8ae02648a2e946c18cbb5f2d52ce375a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.063719 4819 raft_consensus.cc:2466] T e60b4d53c40c4dffab928f0e3fb4a80b P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:38.065243 4912 leader_election.cc:304] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:38.066268 5615 raft_consensus.cc:2747] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:38.070394 5410 tablet_bootstrap.cc:492] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:38.071015 5410 ts_tablet_manager.cc:1397] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.494s user 0.013s sys 0.003s
I20250626 19:57:38.071605 5615 consensus_queue.cc:1035] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:57:38.073479 4912 leader_election.cc:343] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:38.073573 5410 raft_consensus.cc:357] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.074148 5410 raft_consensus.cc:383] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.074416 5410 raft_consensus.cc:738] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:38.084467 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 5bf4c4596b4441839c82b5fe5c5eea37 (DEFAULT_TABLE table=default.Table_092532fc216f4744b12102be8acf3b86 [id=046d6ad2111545a1863ea05a8f75cd4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.085479 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bf4c4596b4441839c82b5fe5c5eea37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.087489 4648 tablet_service.cc:1468] Processing CreateTablet for tablet 781f8ee8c8ad49c6b2391fcdf20bdfdb (DEFAULT_TABLE table=default.Table_c62b1259fcee4798bbcc9a86437cf004 [id=eb61abd19bed46288250e4a8f02c39c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.088073 4942 tablet_service.cc:1468] Processing CreateTablet for tablet 609d36d46bb84017b2a3a7ad48bfccfc (DEFAULT_TABLE table=default.Table_d23ee99af7b845b78d2200b962f681a6 [id=fc725d6df6774d4e922e2d98d461b111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.087894 5410 consensus_queue.cc:260] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.089114 4942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 609d36d46bb84017b2a3a7ad48bfccfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.089242 4796 tablet_service.cc:1468] Processing CreateTablet for tablet 781f8ee8c8ad49c6b2391fcdf20bdfdb (DEFAULT_TABLE table=default.Table_c62b1259fcee4798bbcc9a86437cf004 [id=eb61abd19bed46288250e4a8f02c39c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.090461 4651 tablet_service.cc:1468] Processing CreateTablet for tablet 5bf4c4596b4441839c82b5fe5c5eea37 (DEFAULT_TABLE table=default.Table_092532fc216f4744b12102be8acf3b86 [id=046d6ad2111545a1863ea05a8f75cd4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.090900 4795 tablet_service.cc:1468] Processing CreateTablet for tablet 5bf4c4596b4441839c82b5fe5c5eea37 (DEFAULT_TABLE table=default.Table_092532fc216f4744b12102be8acf3b86 [id=046d6ad2111545a1863ea05a8f75cd4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.092543 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 609d36d46bb84017b2a3a7ad48bfccfc (DEFAULT_TABLE table=default.Table_d23ee99af7b845b78d2200b962f681a6 [id=fc725d6df6774d4e922e2d98d461b111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.095692 4911 leader_election.cc:304] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:38.096407 5615 raft_consensus.cc:2802] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:38.096812 5615 raft_consensus.cc:695] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:38.097506 5615 consensus_queue.cc:237] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.102077 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61fcd55f6d7b4a369383d4f73d5f483d" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:38.107232 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69bd106066b6492bb7ea03d15a18df00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.116076 5366 tablet_bootstrap.cc:492] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:38.116488 5366 ts_tablet_manager.cc:1397] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.143s user 0.011s sys 0.000s
I20250626 19:57:38.118628 5366 raft_consensus.cc:357] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.123528 5366 raft_consensus.cc:383] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.123780 5366 raft_consensus.cc:738] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:38.124395 5366 consensus_queue.cc:260] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.126981 5366 ts_tablet_manager.cc:1428] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250626 19:57:38.127283 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3143181fae16411888d74f9152f3d5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.130177 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bf4c4596b4441839c82b5fe5c5eea37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.131004 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 609d36d46bb84017b2a3a7ad48bfccfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.135154 4796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781f8ee8c8ad49c6b2391fcdf20bdfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.139436 5366 tablet_bootstrap.cc:492] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:38.143447 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 64) took 1984 ms. Trace:
I20250626 19:57:38.143842 5438 rpcz_store.cc:276] 0626 19:57:36.159402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.159486 (+ 84us) service_pool.cc:225] Handling call
0626 19:57:37.726081 (+1566595us) ts_tablet_manager.cc:1145] Deleting tablet a31f1a7739544706bff2a500d6026086
0626 19:57:37.726116 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:37.842707 (+116591us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:38.008946 (+166239us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:38.143384 (+134438us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1566156}
I20250626 19:57:38.145151 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 61) took 1987 ms. Trace:
I20250626 19:57:38.145545 5438 rpcz_store.cc:276] 0626 19:57:36.157742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.161407 (+ 3665us) service_pool.cc:225] Handling call
0626 19:57:38.144920 (+1983513us) ts_tablet_manager.cc:1145] Deleting tablet ea7374ad6ba64e46b7df9852fa33f145
0626 19:57:38.144959 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.145104 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1982964}
I20250626 19:57:38.169279 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet a31f1a7739544706bff2a500d6026086 (table default.Table_fc5a0fda8b274c989cb3b9189b6a838d [id=ee7713e5a64e4b5b8e32bc5862786d05]) successfully deleted
I20250626 19:57:38.177450 5410 ts_tablet_manager.cc:1428] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.106s user 0.016s sys 0.003s
I20250626 19:57:38.184727 5624 raft_consensus.cc:491] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:38.185360 5624 raft_consensus.cc:513] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.187304 5624 leader_election.cc:290] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:38.189360 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 68) took 1817 ms. Trace:
I20250626 19:57:38.189714 5433 rpcz_store.cc:276] 0626 19:57:36.372070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.376887 (+ 4817us) service_pool.cc:225] Handling call
0626 19:57:37.614210 (+1237323us) ts_tablet_manager.cc:1145] Deleting tablet a31f1a7739544706bff2a500d6026086
0626 19:57:37.614248 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:37.911614 (+297366us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:38.038646 (+127032us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:38.189262 (+150616us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1235023}
I20250626 19:57:38.190979 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 72) took 1666 ms. Trace:
I20250626 19:57:38.191236 5433 rpcz_store.cc:276] 0626 19:57:36.524458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.524598 (+ 140us) service_pool.cc:225] Handling call
0626 19:57:38.190682 (+1666084us) ts_tablet_manager.cc:1145] Deleting tablet afd1fafb53434957b002fad5c77431d4
0626 19:57:38.190776 (+ 94us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.190935 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1591328}
I20250626 19:57:38.192252 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 73) took 1666 ms. Trace:
I20250626 19:57:38.192811 4649 tablet_service.cc:1468] Processing CreateTablet for tablet 609d36d46bb84017b2a3a7ad48bfccfc (DEFAULT_TABLE table=default.Table_d23ee99af7b845b78d2200b962f681a6 [id=fc725d6df6774d4e922e2d98d461b111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.193509 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69bd106066b6492bb7ea03d15a18df00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.192533 5433 rpcz_store.cc:276] 0626 19:57:36.525740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.525870 (+ 130us) service_pool.cc:225] Handling call
0626 19:57:38.191988 (+1666118us) ts_tablet_manager.cc:1145] Deleting tablet 5785508e37624f02bc2f6cc93007e3f8
0626 19:57:38.192045 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.192211 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1592470}
I20250626 19:57:38.194470 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 71) took 1795 ms. Trace:
I20250626 19:57:38.194768 5433 rpcz_store.cc:276] 0626 19:57:36.398696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.407301 (+ 8605us) service_pool.cc:225] Handling call
0626 19:57:38.194217 (+1786916us) ts_tablet_manager.cc:1145] Deleting tablet ea7374ad6ba64e46b7df9852fa33f145
0626 19:57:38.194268 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.194428 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1560436}
I20250626 19:57:38.195952 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 75) took 1563 ms. Trace:
I20250626 19:57:38.196225 5433 rpcz_store.cc:276] 0626 19:57:36.632993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.633096 (+ 103us) service_pool.cc:225] Handling call
0626 19:57:38.195658 (+1562562us) ts_tablet_manager.cc:1145] Deleting tablet 512dc911bfce49b29c6fc8cd90523723
0626 19:57:38.195707 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.195895 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1557428}
I20250626 19:57:38.197203 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 74) took 1668 ms. Trace:
I20250626 19:57:38.197459 5433 rpcz_store.cc:276] 0626 19:57:36.532392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.533680 (+ 1288us) service_pool.cc:225] Handling call
0626 19:57:38.196934 (+1663254us) ts_tablet_manager.cc:1145] Deleting tablet 78a8f494f8614e7ebb8ca880b6c80e78
0626 19:57:38.196999 (+ 65us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.197161 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1558564}
I20250626 19:57:38.198433 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 76) took 1559 ms. Trace:
I20250626 19:57:38.199070 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 609d36d46bb84017b2a3a7ad48bfccfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.199759 4651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bf4c4596b4441839c82b5fe5c5eea37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.203133 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3143181fae16411888d74f9152f3d5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.207293 4648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781f8ee8c8ad49c6b2391fcdf20bdfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.211607 5410 tablet_bootstrap.cc:492] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:38.198702 5433 rpcz_store.cc:276] 0626 19:57:36.639752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.640216 (+ 464us) service_pool.cc:225] Handling call
0626 19:57:38.198172 (+1557956us) ts_tablet_manager.cc:1145] Deleting tablet af58f34e2248432fae5c084bd3388028
0626 19:57:38.198225 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.198393 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1027,"tablet-delete.queue_time_us":1557477}
I20250626 19:57:38.215349 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 77) took 1572 ms. Trace:
I20250626 19:57:38.215658 5433 rpcz_store.cc:276] 0626 19:57:36.642868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.678145 (+ 35277us) service_pool.cc:225] Handling call
0626 19:57:38.215037 (+1536892us) ts_tablet_manager.cc:1145] Deleting tablet 4d8ffa9b658c413fa06d02ee6635dd1b
0626 19:57:38.215097 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.215293 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1535305}
I20250626 19:57:38.151468 5366 tablet_bootstrap.cc:654] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:38.216383 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 67) took 2017 ms. Trace:
I20250626 19:57:38.216699 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 78) took 1507 ms. Trace:
I20250626 19:57:38.216722 5438 rpcz_store.cc:276] 0626 19:57:36.199278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.200004 (+ 726us) service_pool.cc:225] Handling call
0626 19:57:38.216140 (+2016136us) ts_tablet_manager.cc:1145] Deleting tablet 1e453a4f8ca44f4a940928d9f4fcd5be
0626 19:57:38.216182 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.216332 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2015484}
I20250626 19:57:38.217001 5433 rpcz_store.cc:276] 0626 19:57:36.708878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.709013 (+ 135us) service_pool.cc:225] Handling call
0626 19:57:38.216434 (+1507421us) ts_tablet_manager.cc:1145] Deleting tablet 1e453a4f8ca44f4a940928d9f4fcd5be
0626 19:57:38.216479 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.216654 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1506819}
I20250626 19:57:38.217805 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 68) took 2018 ms. Trace:
I20250626 19:57:38.218066 5438 rpcz_store.cc:276] 0626 19:57:36.199548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.209835 (+ 10287us) service_pool.cc:225] Handling call
0626 19:57:38.217590 (+2007755us) ts_tablet_manager.cc:1145] Deleting tablet 4d8ffa9b658c413fa06d02ee6635dd1b
0626 19:57:38.217630 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.217760 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2007132}
I20250626 19:57:38.219184 5569 consensus_queue.cc:1035] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:38.219316 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 69) took 1963 ms. Trace:
I20250626 19:57:38.219812 5438 rpcz_store.cc:276] 0626 19:57:36.256139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.256391 (+ 252us) service_pool.cc:225] Handling call
0626 19:57:38.218813 (+1962422us) ts_tablet_manager.cc:1145] Deleting tablet 78a8f494f8614e7ebb8ca880b6c80e78
0626 19:57:38.218864 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.219253 (+ 389us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1919607}
I20250626 19:57:38.222066 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fb3692f4ddb4ff89c1817da4a5cb927" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:38.222174 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet ea7374ad6ba64e46b7df9852fa33f145 because the tablet was not found. No further retry: Not found: Tablet not found: ea7374ad6ba64e46b7df9852fa33f145
W20250626 19:57:38.223062 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 1e453a4f8ca44f4a940928d9f4fcd5be because the tablet was not found. No further retry: Not found: Tablet not found: 1e453a4f8ca44f4a940928d9f4fcd5be
W20250626 19:57:38.223976 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 4d8ffa9b658c413fa06d02ee6635dd1b because the tablet was not found. No further retry: Not found: Tablet not found: 4d8ffa9b658c413fa06d02ee6635dd1b
W20250626 19:57:38.224980 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 78a8f494f8614e7ebb8ca880b6c80e78 because the tablet was not found. No further retry: Not found: Tablet not found: 78a8f494f8614e7ebb8ca880b6c80e78
I20250626 19:57:38.229322 5353 tablet_bootstrap.cc:492] T 419c4018256546ef9431ac2db619ee33 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:38.229782 5353 ts_tablet_manager.cc:1397] T 419c4018256546ef9431ac2db619ee33 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.185s user 0.020s sys 0.031s
I20250626 19:57:38.232654 5506 raft_consensus.cc:491] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:38.233136 5506 raft_consensus.cc:513] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.235158 5506 leader_election.cc:290] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:38.237053 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f834922c9de46dca48c47748913b11a" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:38.237675 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:38.238215 5353 raft_consensus.cc:357] T 419c4018256546ef9431ac2db619ee33 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.239001 5353 raft_consensus.cc:383] T 419c4018256546ef9431ac2db619ee33 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.239254 5353 raft_consensus.cc:738] T 419c4018256546ef9431ac2db619ee33 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:38.239893 5353 consensus_queue.cc:260] T 419c4018256546ef9431ac2db619ee33 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.241912 5353 ts_tablet_manager.cc:1428] T 419c4018256546ef9431ac2db619ee33 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20250626 19:57:38.242802 5353 tablet_bootstrap.cc:492] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:38.244341 5410 tablet_bootstrap.cc:654] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:38.247081 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 70) took 1846 ms. Trace:
I20250626 19:57:38.247422 5438 rpcz_store.cc:276] 0626 19:57:36.400745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.400913 (+ 168us) service_pool.cc:225] Handling call
0626 19:57:38.220800 (+1819887us) ts_tablet_manager.cc:1145] Deleting tablet 82823fd9ceb1485386616319fa7aeb43
0626 19:57:38.220850 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.247019 (+ 26169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1818709}
I20250626 19:57:38.255091 5438 tablet_replica.cc:331] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:38.255853 5438 raft_consensus.cc:2241] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:38.256560 5438 raft_consensus.cc:2270] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:38.256786 5353 tablet_bootstrap.cc:654] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:38.257522 4674 raft_consensus.cc:2466] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 2.
W20250626 19:57:38.254701 4383 catalog_manager.cc:4908] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 82823fd9ceb1485386616319fa7aeb43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82823fd9ceb1485386616319fa7aeb43 already in progress: creating tablet
I20250626 19:57:38.239199 4390 catalog_manager.cc:5582] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:38.128319 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fb3692f4ddb4ff89c1817da4a5cb927" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:38.189428 5642 raft_consensus.cc:491] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:38.263159 5642 raft_consensus.cc:513] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.265043 5642 leader_election.cc:290] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:38.266026 4819 raft_consensus.cc:2466] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:38.263213 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet a31f1a7739544706bff2a500d6026086 (table default.Table_fc5a0fda8b274c989cb3b9189b6a838d [id=ee7713e5a64e4b5b8e32bc5862786d05]) successfully deleted
W20250626 19:57:38.269029 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet afd1fafb53434957b002fad5c77431d4 because the tablet was not found. No further retry: Not found: Tablet not found: afd1fafb53434957b002fad5c77431d4
W20250626 19:57:38.269630 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 5785508e37624f02bc2f6cc93007e3f8 because the tablet was not found. No further retry: Not found: Tablet not found: 5785508e37624f02bc2f6cc93007e3f8
W20250626 19:57:38.270156 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet ea7374ad6ba64e46b7df9852fa33f145 because the tablet was not found. No further retry: Not found: Tablet not found: ea7374ad6ba64e46b7df9852fa33f145
W20250626 19:57:38.270715 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 512dc911bfce49b29c6fc8cd90523723 because the tablet was not found. No further retry: Not found: Tablet not found: 512dc911bfce49b29c6fc8cd90523723
I20250626 19:57:38.273017 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:38.272974 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "419c4018256546ef9431ac2db619ee33" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:38.273500 4978 raft_consensus.cc:2466] T 419c4018256546ef9431ac2db619ee33 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:38.273658 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65af5ae255d74143976392bccc50f3ba" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:38.274310 4979 raft_consensus.cc:2391] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 93f885c6802341c889a40e9c20fbfff1 in this term.
I20250626 19:57:38.276959 5353 tablet_bootstrap.cc:492] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:38.277319 5353 ts_tablet_manager.cc:1397] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.000s
I20250626 19:57:38.277916 4911 leader_election.cc:304] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:38.278844 5438 ts_tablet_manager.cc:1905] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:38.279573 5615 raft_consensus.cc:2802] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:38.279901 5615 raft_consensus.cc:491] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:38.279716 5353 raft_consensus.cc:357] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.280251 5615 raft_consensus.cc:3058] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.280360 5353 raft_consensus.cc:383] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.280632 5353 raft_consensus.cc:738] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:38.281280 5353 consensus_queue.cc:260] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:38.277527 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 78a8f494f8614e7ebb8ca880b6c80e78 because the tablet was not found. No further retry: Not found: Tablet not found: 78a8f494f8614e7ebb8ca880b6c80e78
W20250626 19:57:38.283288 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet af58f34e2248432fae5c084bd3388028 because the tablet was not found. No further retry: Not found: Tablet not found: af58f34e2248432fae5c084bd3388028
W20250626 19:57:38.283859 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 4d8ffa9b658c413fa06d02ee6635dd1b because the tablet was not found. No further retry: Not found: Tablet not found: 4d8ffa9b658c413fa06d02ee6635dd1b
W20250626 19:57:38.284404 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 1e453a4f8ca44f4a940928d9f4fcd5be because the tablet was not found. No further retry: Not found: Tablet not found: 1e453a4f8ca44f4a940928d9f4fcd5be
I20250626 19:57:38.286170 4390 catalog_manager.cc:5582] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } }
W20250626 19:57:38.289613 4912 leader_election.cc:343] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:38.293128 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 3c03691d021e4ee19d5bb287ab633a3c (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:38.294037 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:38.295060 5353 ts_tablet_manager.cc:1428] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20250626 19:57:38.296144 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:38.297936 5353 tablet_bootstrap.cc:492] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:38.311161 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65af5ae255d74143976392bccc50f3ba" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
W20250626 19:57:38.281184 4756 leader_election.cc:343] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:38.313731 4756 leader_election.cc:304] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:38.319386 5642 raft_consensus.cc:2802] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:38.320096 5642 raft_consensus.cc:491] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:38.320438 5642 raft_consensus.cc:3058] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.323319 5569 raft_consensus.cc:491] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:38.323742 5569 raft_consensus.cc:513] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.325490 5569 leader_election.cc:290] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:38.326622 4386 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5c113e03594b428a979435618fe373c6 consensus state has no leader
I20250626 19:57:38.330935 5353 tablet_bootstrap.cc:654] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:38.246541 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e60b4d53c40c4dffab928f0e3fb4a80b" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:38.284853 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f834922c9de46dca48c47748913b11a" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:38.335071 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7facb77eb579448f9533d285cba8650e" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:38.292075 4672 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "419c4018256546ef9431ac2db619ee33" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:38.296311 5410 tablet_bootstrap.cc:492] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:38.336566 5410 ts_tablet_manager.cc:1397] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.125s user 0.009s sys 0.000s
I20250626 19:57:38.338618 5366 tablet_bootstrap.cc:492] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:38.339066 5366 ts_tablet_manager.cc:1397] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.200s user 0.011s sys 0.005s
I20250626 19:57:38.338742 5410 raft_consensus.cc:357] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.339371 5410 raft_consensus.cc:383] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.339645 5410 raft_consensus.cc:738] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:38.340183 5410 consensus_queue.cc:260] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.342677 5410 ts_tablet_manager.cc:1428] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20250626 19:57:38.343921 5642 raft_consensus.cc:513] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.347021 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "419c4018256546ef9431ac2db619ee33" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:38.347560 4978 raft_consensus.cc:3058] T 419c4018256546ef9431ac2db619ee33 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.354554 4978 raft_consensus.cc:2466] T 419c4018256546ef9431ac2db619ee33 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:38.355681 4756 leader_election.cc:304] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:38.356105 4979 raft_consensus.cc:3058] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.356431 5506 raft_consensus.cc:2802] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:38.356837 5506 raft_consensus.cc:695] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:38.357067 5353 tablet_bootstrap.cc:492] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:38.357436 5353 ts_tablet_manager.cc:1397] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.004s
I20250626 19:57:38.357553 5506 consensus_queue.cc:237] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.359565 5353 raft_consensus.cc:357] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.360124 5353 raft_consensus.cc:383] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.360356 5353 raft_consensus.cc:738] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
W20250626 19:57:38.360636 4755 leader_election.cc:343] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:38.361030 4755 leader_election.cc:304] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:38.361722 5429 raft_consensus.cc:2747] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:38.362406 5353 consensus_queue.cc:260] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:38.362910 4755 leader_election.cc:343] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:38.341361 5366 raft_consensus.cc:357] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.367866 5366 raft_consensus.cc:383] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.368808 5366 raft_consensus.cc:738] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:38.370388 5410 tablet_bootstrap.cc:492] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:38.371150 5642 leader_election.cc:290] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:38.372288 5366 consensus_queue.cc:260] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.375325 5366 ts_tablet_manager.cc:1428] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.036s user 0.005s sys 0.001s
I20250626 19:57:38.376554 5353 ts_tablet_manager.cc:1428] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.019s user 0.004s sys 0.002s
I20250626 19:57:38.384397 4392 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 3c03691d021e4ee19d5bb287ab633a3c
I20250626 19:57:38.385818 4392 catalog_manager.cc:6652] ReplaceTablet: tablet 3c03691d021e4ee19d5bb287ab633a3c deleted and replaced by tablet de0724ba71824a5daaf6f3708b4431b7
I20250626 19:57:38.386691 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c03691d021e4ee19d5bb287ab633a3c with delete_type TABLET_DATA_DELETED (replaced by tablet de0724ba71824a5daaf6f3708b4431b7) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:38.388036 5366 tablet_bootstrap.cc:492] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:38.394088 5353 tablet_bootstrap.cc:492] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:38.351322 4609 leader_election.cc:304] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
W20250626 19:57:38.401022 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 49893.2us
I20250626 19:57:38.368988 4674 raft_consensus.cc:3058] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.402298 5353 tablet_bootstrap.cc:654] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:38.217897 5433 tablet_replica.cc:331] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:38.385360 4672 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "419c4018256546ef9431ac2db619ee33" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:38.407248 5433 raft_consensus.cc:2241] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250626 19:57:38.407994 4755 leader_election.cc:343] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:38.411573 5623 raft_consensus.cc:2747] T 61fcd55f6d7b4a369383d4f73d5f483d P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:38.413162 5433 raft_consensus.cc:2270] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:38.415323 5433 ts_tablet_manager.cc:1905] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:38.420336 5366 tablet_bootstrap.cc:654] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:38.428835 5353 tablet_bootstrap.cc:492] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:38.429222 5353 ts_tablet_manager.cc:1397] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.000s
I20250626 19:57:38.421963 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c03691d021e4ee19d5bb287ab633a3c with delete_type TABLET_DATA_DELETED (replaced by tablet de0724ba71824a5daaf6f3708b4431b7) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:38.431545 5353 raft_consensus.cc:357] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.432134 5353 raft_consensus.cc:383] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.432397 5353 raft_consensus.cc:738] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:38.432974 5353 consensus_queue.cc:260] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.444353 4799 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c03691d021e4ee19d5bb287ab633a3c with delete_type TABLET_DATA_DELETED (replaced by tablet de0724ba71824a5daaf6f3708b4431b7) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:38.446152 5353 ts_tablet_manager.cc:1428] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20250626 19:57:38.447539 4609 leader_election.cc:304] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:38.451452 5653 tablet_replica.cc:331] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:38.452519 5653 raft_consensus.cc:2241] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:38.453368 5653 raft_consensus.cc:2270] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:38.458058 5353 tablet_bootstrap.cc:492] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:38.461035 5623 raft_consensus.cc:2747] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:38.463946 5438 ts_tablet_manager.cc:1918] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:38.464277 5438 log.cc:1199] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/2ab0320af47049cca3ca87172b9fe5e5
I20250626 19:57:38.464924 5438 ts_tablet_manager.cc:1939] T 2ab0320af47049cca3ca87172b9fe5e5 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:38.468225 4391 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:38.468907 4391 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
W20250626 19:57:38.470281 4912 leader_election.cc:343] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:38.478152 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 72) took 1737 ms. Trace:
I20250626 19:57:38.478173 5623 raft_consensus.cc:491] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:38.470945 5429 consensus_queue.cc:1035] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20250626 19:57:38.478806 5623 raft_consensus.cc:513] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.479547 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7facb77eb579448f9533d285cba8650e" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:38.480686 5623 leader_election.cc:290] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:38.484534 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94cefa59de044a36a2afaaf5531128d8" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:38.485128 4819 raft_consensus.cc:2466] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:38.487074 4609 leader_election.cc:304] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:38.487573 5438 rpcz_store.cc:276] 0626 19:57:36.740327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.740432 (+ 105us) service_pool.cc:225] Handling call
0626 19:57:38.254933 (+1514501us) ts_tablet_manager.cc:1145] Deleting tablet 2ab0320af47049cca3ca87172b9fe5e5
0626 19:57:38.254978 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.303050 (+ 48072us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:38.463723 (+160673us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:38.478090 (+ 14367us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11439,"tablet-delete.queue_time_us":1506288}
I20250626 19:57:38.488881 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 73) took 1476 ms. Trace:
I20250626 19:57:38.489179 5438 rpcz_store.cc:276] 0626 19:57:37.012965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:37.013200 (+ 235us) service_pool.cc:225] Handling call
0626 19:57:38.488656 (+1475456us) ts_tablet_manager.cc:1145] Deleting tablet 5785508e37624f02bc2f6cc93007e3f8
0626 19:57:38.488693 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.488832 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1474837}
I20250626 19:57:38.490288 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 75) took 1329 ms. Trace:
I20250626 19:57:38.490597 5438 rpcz_store.cc:276] 0626 19:57:37.160470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:37.160599 (+ 129us) service_pool.cc:225] Handling call
0626 19:57:38.489989 (+1329390us) ts_tablet_manager.cc:1145] Deleting tablet 36997dab0ab3430bba63a90805759b30
0626 19:57:38.490029 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.490239 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1204125}
I20250626 19:57:38.495642 5623 raft_consensus.cc:2802] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:38.496014 5623 raft_consensus.cc:491] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:38.496394 5623 raft_consensus.cc:3058] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.489073 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 2ab0320af47049cca3ca87172b9fe5e5 (table default.Table_d23ee99af7b845b78d2200b962f681a6 [id=fc725d6df6774d4e922e2d98d461b111]) successfully deleted
I20250626 19:57:38.498493 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94cefa59de044a36a2afaaf5531128d8" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:38.498626 4912 leader_election.cc:304] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:38.499002 4978 raft_consensus.cc:2466] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
W20250626 19:57:38.499265 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 5785508e37624f02bc2f6cc93007e3f8 because the tablet was not found. No further retry: Not found: Tablet not found: 5785508e37624f02bc2f6cc93007e3f8
W20250626 19:57:38.499878 4383 catalog_manager.cc:4908] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 36997dab0ab3430bba63a90805759b30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36997dab0ab3430bba63a90805759b30 already in progress: creating tablet
I20250626 19:57:38.501027 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:38.502133 4391 catalog_manager.cc:5582] T 7facb77eb579448f9533d285cba8650e P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:38.504290 4391 catalog_manager.cc:5582] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:38.505980 5366 tablet_bootstrap.cc:492] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:38.508406 5410 tablet_bootstrap.cc:654] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:38.511111 5366 ts_tablet_manager.cc:1397] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.124s user 0.011s sys 0.006s
I20250626 19:57:38.514333 5366 raft_consensus.cc:357] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.520650 4391 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:38.528492 5569 raft_consensus.cc:491] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:38.529346 5569 raft_consensus.cc:513] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.531154 5569 leader_election.cc:290] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:38.533452 5623 raft_consensus.cc:513] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.515127 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 76) took 1305 ms. Trace:
I20250626 19:57:38.519413 5642 raft_consensus.cc:491] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:38.537472 5366 raft_consensus.cc:383] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.537858 5366 raft_consensus.cc:738] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:38.537585 5642 raft_consensus.cc:513] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.537441 5438 rpcz_store.cc:276] 0626 19:57:37.209303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:37.210012 (+ 709us) service_pool.cc:225] Handling call
0626 19:57:38.491465 (+1281453us) ts_tablet_manager.cc:1145] Deleting tablet cb34ffcd4da04e32a1220b3f298ec22d
0626 19:57:38.514804 (+ 23339us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.515075 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1195857}
I20250626 19:57:38.539506 5642 leader_election.cc:290] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:38.532428 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f834922c9de46dca48c47748913b11a" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:38.540563 5438 tablet_replica.cc:331] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:38.540961 4819 raft_consensus.cc:2466] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:38.541234 5623 leader_election.cc:290] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:38.541452 5438 raft_consensus.cc:2241] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:38.542219 5438 raft_consensus.cc:2270] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:38.542150 4911 leader_election.cc:304] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:38.542477 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94cefa59de044a36a2afaaf5531128d8" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:38.542991 4819 raft_consensus.cc:3058] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.544437 5438 ts_tablet_manager.cc:1905] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:38.544947 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c86ced5a3dc48e380244a9548d2f4c8" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:38.546115 4755 leader_election.cc:343] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:38.540166 4383 catalog_manager.cc:4908] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet cb34ffcd4da04e32a1220b3f298ec22d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb34ffcd4da04e32a1220b3f298ec22d already in progress: creating tablet
I20250626 19:57:38.526222 4391 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:38.551141 4819 raft_consensus.cc:2466] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:38.551839 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:38.553041 4609 leader_election.cc:304] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:38.540232 5366 consensus_queue.cc:260] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.556816 5623 raft_consensus.cc:2802] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:38.558260 5623 raft_consensus.cc:695] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:38.558701 5366 ts_tablet_manager.cc:1428] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20250626 19:57:38.565469 5623 consensus_queue.cc:237] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.568202 5366 tablet_bootstrap.cc:492] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:38.583067 5366 tablet_bootstrap.cc:654] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:38.330335 5615 raft_consensus.cc:513] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.603261 5615 leader_election.cc:290] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:38.611784 5615 raft_consensus.cc:2802] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:38.610345 5353 tablet_bootstrap.cc:654] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:38.613911 5653 ts_tablet_manager.cc:1905] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:38.625025 5366 tablet_bootstrap.cc:492] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:38.625583 5366 ts_tablet_manager.cc:1397] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.000s
I20250626 19:57:38.629698 5366 raft_consensus.cc:357] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.630576 5366 raft_consensus.cc:383] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.630915 5366 raft_consensus.cc:738] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:38.631680 5366 consensus_queue.cc:260] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.633911 5366 ts_tablet_manager.cc:1428] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250626 19:57:38.635705 5366 tablet_bootstrap.cc:492] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:38.645993 5438 ts_tablet_manager.cc:1918] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:38.646459 5438 log.cc:1199] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/7391b9eaea3e426ba4118fc8b53b72a6
I20250626 19:57:38.648609 5366 tablet_bootstrap.cc:654] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:38.652245 5410 tablet_bootstrap.cc:492] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:38.652729 5410 ts_tablet_manager.cc:1397] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.283s user 0.003s sys 0.006s
I20250626 19:57:38.663632 5438 ts_tablet_manager.cc:1939] T 7391b9eaea3e426ba4118fc8b53b72a6 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:38.665551 5438 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 77) took 1009 ms. Trace:
W20250626 19:57:38.672185 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 173693us
I20250626 19:57:38.667913 5366 tablet_bootstrap.cc:492] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:38.673964 5366 ts_tablet_manager.cc:1397] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.039s user 0.020s sys 0.005s
I20250626 19:57:38.674432 5429 consensus_queue.cc:1035] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:38.677484 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fb3692f4ddb4ff89c1817da4a5cb927" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:38.673748 5438 rpcz_store.cc:276] 0626 19:57:37.655839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:37.656642 (+ 803us) service_pool.cc:225] Handling call
0626 19:57:38.540392 (+883750us) ts_tablet_manager.cc:1145] Deleting tablet 7391b9eaea3e426ba4118fc8b53b72a6
0626 19:57:38.540446 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.613214 (+ 72768us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:38.645792 (+ 32578us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:38.665490 (+ 19698us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":883290}
I20250626 19:57:38.677666 5366 raft_consensus.cc:357] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.678263 5366 raft_consensus.cc:383] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.678510 5366 raft_consensus.cc:738] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:38.678498 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 7391b9eaea3e426ba4118fc8b53b72a6 (table default.Table_e5f9396b41eb4c5eb98ec42ec48d7cdc [id=bad1d85ef3a347ab9abe78289306ca4d]) successfully deleted
I20250626 19:57:38.679620 5438 tablet_replica.cc:331] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:38.680030 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f834922c9de46dca48c47748913b11a" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:38.680358 5438 raft_consensus.cc:2241] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:38.680814 5438 raft_consensus.cc:2270] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:38.682487 5664 raft_consensus.cc:491] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:38.682955 5664 raft_consensus.cc:513] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.684731 5664 leader_election.cc:290] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:38.692586 5438 ts_tablet_manager.cc:1905] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:38.694208 5433 ts_tablet_manager.cc:1918] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:38.694576 5433 log.cc:1199] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/2ab0320af47049cca3ca87172b9fe5e5
I20250626 19:57:38.697993 5429 raft_consensus.cc:491] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:38.698565 5429 raft_consensus.cc:513] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.700155 5429 leader_election.cc:290] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:38.615434 5615 raft_consensus.cc:491] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:38.702821 5615 raft_consensus.cc:3058] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:38.727432 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 185427us
I20250626 19:57:38.728957 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fb3692f4ddb4ff89c1817da4a5cb927" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:38.729491 4819 raft_consensus.cc:3058] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.710366 5615 raft_consensus.cc:513] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.736729 4819 raft_consensus.cc:2466] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:38.735080 5569 raft_consensus.cc:2747] T 7facb77eb579448f9533d285cba8650e P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:38.739426 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c86ced5a3dc48e380244a9548d2f4c8" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:38.740113 4819 raft_consensus.cc:2466] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:38.742974 5569 raft_consensus.cc:491] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:38.743413 5569 raft_consensus.cc:513] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.743609 4911 leader_election.cc:304] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:38.744608 4911 leader_election.cc:304] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:38.745172 5569 leader_election.cc:290] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:38.745640 5660 raft_consensus.cc:2802] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:38.746142 5660 raft_consensus.cc:695] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:38.746984 5668 raft_consensus.cc:2802] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:38.747356 5668 raft_consensus.cc:491] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:38.747696 5668 raft_consensus.cc:3058] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.749413 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82823fd9ceb1485386616319fa7aeb43" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:38.746999 5660 consensus_queue.cc:237] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.750090 4819 raft_consensus.cc:2466] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:38.751696 4911 leader_election.cc:304] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:38.752599 5615 leader_election.cc:290] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:38.757802 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f834922c9de46dca48c47748913b11a" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:38.758406 4819 raft_consensus.cc:3058] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.760277 5353 tablet_bootstrap.cc:492] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:38.760687 5353 ts_tablet_manager.cc:1397] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.304s user 0.007s sys 0.001s
I20250626 19:57:38.679086 5366 consensus_queue.cc:260] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.765905 5366 ts_tablet_manager.cc:1428] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.092s user 0.005s sys 0.000s
I20250626 19:57:38.766724 5366 tablet_bootstrap.cc:492] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:38.774449 4819 raft_consensus.cc:2466] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:38.777956 4911 leader_election.cc:304] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:38.760691 5668 raft_consensus.cc:513] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.787578 5438 ts_tablet_manager.cc:1918] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:38.787987 5438 log.cc:1199] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/3c03691d021e4ee19d5bb287ab633a3c
I20250626 19:57:38.788722 5438 ts_tablet_manager.cc:1939] T 3c03691d021e4ee19d5bb287ab633a3c P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:38.785405 5668 leader_election.cc:290] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:38.792831 5353 raft_consensus.cc:357] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.794042 5353 raft_consensus.cc:383] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.794454 5353 raft_consensus.cc:738] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:38.795478 5615 raft_consensus.cc:2802] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:38.795918 5615 raft_consensus.cc:695] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:38.796108 5353 consensus_queue.cc:260] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.797685 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c86ced5a3dc48e380244a9548d2f4c8" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:38.796628 5615 consensus_queue.cc:237] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.798801 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94cefa59de044a36a2afaaf5531128d8" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:38.799300 4979 raft_consensus.cc:3058] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.799424 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c86ced5a3dc48e380244a9548d2f4c8" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:38.800062 4978 raft_consensus.cc:2391] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93f885c6802341c889a40e9c20fbfff1 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:38.797076 5664 raft_consensus.cc:2802] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:38.805095 5664 raft_consensus.cc:491] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:38.805871 5506 raft_consensus.cc:491] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:38.807401 5506 raft_consensus.cc:513] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.809906 5506 leader_election.cc:290] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:38.806712 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b8473259485460783f62f8193d5b02c" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:38.814548 4819 raft_consensus.cc:3058] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.814585 4978 raft_consensus.cc:2466] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:38.814141 5664 raft_consensus.cc:3058] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.816294 5353 ts_tablet_manager.cc:1428] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.055s user 0.005s sys 0.000s
I20250626 19:57:38.816875 5506 raft_consensus.cc:491] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:38.817435 5506 raft_consensus.cc:513] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.821080 5506 leader_election.cc:290] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:38.821177 4756 leader_election.cc:304] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:38.825446 5353 tablet_bootstrap.cc:492] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
W20250626 19:57:38.827118 4385 catalog_manager.cc:5001] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): alter failed for tablet 3c03691d021e4ee19d5bb287ab633a3c (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]),no further retry: Illegal state: Tablet not RUNNING: STOPPED
W20250626 19:57:38.829075 4912 leader_election.cc:343] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:38.835408 4979 raft_consensus.cc:2466] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:38.701939 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b8473259485460783f62f8193d5b02c" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:38.726881 5410 raft_consensus.cc:357] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.811407 4672 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36997dab0ab3430bba63a90805759b30" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:38.836167 5410 raft_consensus.cc:383] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.816785 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c86ced5a3dc48e380244a9548d2f4c8" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:38.836440 5410 raft_consensus.cc:738] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:38.821254 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7c352224d0a46c6a922bf0a0d765f17" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:38.823479 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82823fd9ceb1485386616319fa7aeb43" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:38.824764 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f834922c9de46dca48c47748913b11a" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:38.837021 5410 consensus_queue.cc:260] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.826432 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c86ced5a3dc48e380244a9548d2f4c8" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
W20250626 19:57:38.838483 4755 leader_election.cc:343] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:38.839082 4755 leader_election.cc:343] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:38.839182 4670 raft_consensus.cc:3058] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.840783 5433 ts_tablet_manager.cc:1939] T 2ab0320af47049cca3ca87172b9fe5e5 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:38.848475 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 81) took 2071 ms. Trace:
I20250626 19:57:38.848886 5433 rpcz_store.cc:276] 0626 19:57:36.777559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.781536 (+ 3977us) service_pool.cc:225] Handling call
0626 19:57:38.217751 (+1436215us) ts_tablet_manager.cc:1145] Deleting tablet 2ab0320af47049cca3ca87172b9fe5e5
0626 19:57:38.217798 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.592447 (+374649us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:38.694003 (+101556us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:38.848420 (+154417us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5313,"tablet-delete.queue_time_us":1435687}
I20250626 19:57:38.850533 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 2ab0320af47049cca3ca87172b9fe5e5 (table default.Table_d23ee99af7b845b78d2200b962f681a6 [id=fc725d6df6774d4e922e2d98d461b111]) successfully deleted
I20250626 19:57:38.851022 5410 ts_tablet_manager.cc:1428] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.198s user 0.004s sys 0.001s
I20250626 19:57:38.851631 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 80) took 2074 ms. Trace:
I20250626 19:57:38.852442 4819 raft_consensus.cc:2466] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:38.866550 5506 raft_consensus.cc:2747] T 6c86ced5a3dc48e380244a9548d2f4c8 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:38.872522 4755 leader_election.cc:343] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:38.867061 4756 leader_election.cc:304] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:38.874548 5506 raft_consensus.cc:2802] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:38.874871 5506 raft_consensus.cc:491] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:38.875191 5506 raft_consensus.cc:3058] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.875315 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 3c03691d021e4ee19d5bb287ab633a3c (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]) successfully deleted
I20250626 19:57:38.893587 5675 tablet.cc:1722] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250626 19:57:38.936697 5366 tablet_bootstrap.cc:654] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:38.948189 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 9172cb6576b240f6b25c3b1267618e6c (DEFAULT_TABLE table=default.Table_2e7897256c7140aeb545751922965c14 [id=441b14db312f4da1be3f2219cf745bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.949139 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9172cb6576b240f6b25c3b1267618e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.848299 5353 tablet_bootstrap.cc:654] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:38.850397 5664 raft_consensus.cc:513] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.956143 5506 raft_consensus.cc:513] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.957175 5664 leader_election.cc:290] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:38.865257 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36997dab0ab3430bba63a90805759b30" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:38.866272 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7c352224d0a46c6a922bf0a0d765f17" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:38.958281 4975 raft_consensus.cc:2466] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:38.870023 4911 leader_election.cc:304] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
W20250626 19:57:38.959303 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 89403.3us
I20250626 19:57:38.960095 5506 leader_election.cc:290] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:38.851922 5410 tablet_bootstrap.cc:492] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:38.960621 5433 rpcz_store.cc:276] 0626 19:57:36.777147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.781693 (+ 4546us) service_pool.cc:225] Handling call
0626 19:57:38.851352 (+2069659us) ts_tablet_manager.cc:1145] Deleting tablet 499f6ae3262940b4859c14a53722482f
0626 19:57:38.851407 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.851584 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2068910}
W20250626 19:57:38.961802 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 499f6ae3262940b4859c14a53722482f because the tablet was not found. No further retry: Not found: Tablet not found: 499f6ae3262940b4859c14a53722482f
I20250626 19:57:38.962994 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 84) took 1782 ms. Trace:
I20250626 19:57:38.859318 4670 raft_consensus.cc:2466] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:38.963308 5433 rpcz_store.cc:276] 0626 19:57:37.180330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:37.180430 (+ 100us) service_pool.cc:225] Handling call
0626 19:57:38.962545 (+1782115us) ts_tablet_manager.cc:1145] Deleting tablet 36997dab0ab3430bba63a90805759b30
0626 19:57:38.962604 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.962918 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1780892}
W20250626 19:57:38.964632 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 36997dab0ab3430bba63a90805759b30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36997dab0ab3430bba63a90805759b30 already in progress: creating tablet
I20250626 19:57:38.965835 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 85) took 1772 ms. Trace:
I20250626 19:57:38.944751 4958 tablet_service.cc:1468] Processing CreateTablet for tablet 9172cb6576b240f6b25c3b1267618e6c (DEFAULT_TABLE table=default.Table_2e7897256c7140aeb545751922965c14 [id=441b14db312f4da1be3f2219cf745bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.891865 5674 tablet.cc:1722] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250626 19:57:38.966141 5433 rpcz_store.cc:276] 0626 19:57:37.193398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:37.193571 (+ 173us) service_pool.cc:225] Handling call
0626 19:57:38.965376 (+1771805us) ts_tablet_manager.cc:1145] Deleting tablet cb34ffcd4da04e32a1220b3f298ec22d
0626 19:57:38.965455 (+ 79us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:38.965785 (+ 330us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1771134}
I20250626 19:57:38.967144 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9172cb6576b240f6b25c3b1267618e6c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:38.967322 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet cb34ffcd4da04e32a1220b3f298ec22d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb34ffcd4da04e32a1220b3f298ec22d already in progress: creating tablet
I20250626 19:57:38.947355 4649 tablet_service.cc:1468] Processing CreateTablet for tablet 9172cb6576b240f6b25c3b1267618e6c (DEFAULT_TABLE table=default.Table_2e7897256c7140aeb545751922965c14 [id=441b14db312f4da1be3f2219cf745bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:38.971953 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9172cb6576b240f6b25c3b1267618e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:38.972784 5664 raft_consensus.cc:2802] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:38.972770 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82823fd9ceb1485386616319fa7aeb43" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:38.973268 5664 raft_consensus.cc:695] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:38.973330 4820 raft_consensus.cc:3058] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:38.974009 5664 consensus_queue.cc:237] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.959625 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b8473259485460783f62f8193d5b02c" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
W20250626 19:57:38.976814 4755 leader_election.cc:343] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:38.979555 5569 consensus_queue.cc:1035] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:38.981757 5353 tablet_bootstrap.cc:492] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:38.982136 5353 ts_tablet_manager.cc:1397] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.157s user 0.005s sys 0.013s
I20250626 19:57:38.983611 4820 raft_consensus.cc:2466] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:38.984700 4911 leader_election.cc:304] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:38.985469 5569 raft_consensus.cc:2802] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:38.985862 5569 raft_consensus.cc:695] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:38.986580 5569 consensus_queue.cc:237] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.988310 5353 raft_consensus.cc:357] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.988896 5353 raft_consensus.cc:383] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:38.989141 5353 raft_consensus.cc:738] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:38.989670 5353 consensus_queue.cc:260] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:38.991856 5410 tablet_bootstrap.cc:654] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:38.948397 5671 log.cc:927] Time spent T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20250626 19:57:39.000640 5353 ts_tablet_manager.cc:1428] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.018s user 0.001s sys 0.005s
I20250626 19:57:39.008044 5353 tablet_bootstrap.cc:492] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:39.013899 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:39.018539 5353 tablet_bootstrap.cc:654] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:39.022043 4950 tablet_service.cc:1468] Processing CreateTablet for tablet d8be634cb36d40dca3f9338dab93ad2c (DEFAULT_TABLE table=default.Table_6dc13f5ceb674e40919de2ca36d9427a [id=160181772a0a40ebac8a3fe697667b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.023192 4950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8be634cb36d40dca3f9338dab93ad2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.026921 5410 tablet_bootstrap.cc:492] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:39.027315 5410 ts_tablet_manager.cc:1397] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.176s user 0.008s sys 0.003s
I20250626 19:57:38.952317 5653 ts_tablet_manager.cc:1918] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:39.028132 5653 log.cc:1199] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/3c03691d021e4ee19d5bb287ab633a3c
I20250626 19:57:39.028887 5653 ts_tablet_manager.cc:1939] T 3c03691d021e4ee19d5bb287ab633a3c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:39.029584 5410 raft_consensus.cc:357] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:38.955420 4912 leader_election.cc:343] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:39.031164 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 75813us
I20250626 19:57:39.030725 5410 raft_consensus.cc:383] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:39.031759 5410 raft_consensus.cc:738] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
W20250626 19:57:39.031769 4912 leader_election.cc:343] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:39.032478 4912 leader_election.cc:343] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:39.033170 4912 leader_election.cc:343] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:39.014680 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:39.047914 5506 raft_consensus.cc:491] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:39.033934 5410 consensus_queue.cc:260] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.048380 5506 raft_consensus.cc:513] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.050288 5506 leader_election.cc:290] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:39.052366 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8122b26848141a39d2462dd3491f9cd" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:39.053529 4755 leader_election.cc:343] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:39.058543 4796 tablet_service.cc:1468] Processing CreateTablet for tablet d8be634cb36d40dca3f9338dab93ad2c (DEFAULT_TABLE table=default.Table_6dc13f5ceb674e40919de2ca36d9427a [id=160181772a0a40ebac8a3fe697667b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.059470 4954 tablet_service.cc:1468] Processing CreateTablet for tablet ef1abf340cc34e00aa67e4588c61c32d (DEFAULT_TABLE table=default.Table_e4d8a4cf39764c5fa002644d60f964b3 [id=d4284cd20f084362a5e9c533f5909b26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.060026 4796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8be634cb36d40dca3f9338dab93ad2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.060345 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1abf340cc34e00aa67e4588c61c32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.068107 5366 tablet_bootstrap.cc:492] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:39.068514 5366 ts_tablet_manager.cc:1397] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.302s user 0.006s sys 0.005s
W20250626 19:57:39.079272 4611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 44328.2us
I20250626 19:57:39.083194 5433 tablet_replica.cc:331] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:39.083600 4675 raft_consensus.cc:1273] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 2 FOLLOWER]: Refusing update from remote peer 93f885c6802341c889a40e9c20fbfff1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250626 19:57:39.084836 5429 consensus_queue.cc:1035] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:39.086843 5433 raft_consensus.cc:2241] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:39.087392 5433 raft_consensus.cc:2270] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:39.089603 5433 ts_tablet_manager.cc:1905] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:39.090622 4756 leader_election.cc:343] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:39.091132 4756 leader_election.cc:304] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:39.092083 4756 leader_election.cc:304] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:39.093111 5642 raft_consensus.cc:2747] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:39.093708 5642 raft_consensus.cc:2802] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:39.094036 5642 raft_consensus.cc:491] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:39.094408 5642 raft_consensus.cc:3058] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:39.097049 5366 raft_consensus.cc:357] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.097671 5366 raft_consensus.cc:383] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:39.097894 5366 raft_consensus.cc:738] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:39.102013 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:39.102126 4976 raft_consensus.cc:1273] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 2 FOLLOWER]: Refusing update from remote peer 93f885c6802341c889a40e9c20fbfff1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250626 19:57:39.103552 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b8473259485460783f62f8193d5b02c" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:39.104224 4976 raft_consensus.cc:3058] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:39.104948 5642 raft_consensus.cc:513] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.109304 5642 leader_election.cc:290] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:39.112483 4976 raft_consensus.cc:2466] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:39.113075 4795 tablet_service.cc:1468] Processing CreateTablet for tablet ef1abf340cc34e00aa67e4588c61c32d (DEFAULT_TABLE table=default.Table_e4d8a4cf39764c5fa002644d60f964b3 [id=d4284cd20f084362a5e9c533f5909b26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.113891 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 3c03691d021e4ee19d5bb287ab633a3c (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]) successfully deleted
I20250626 19:57:39.114136 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1abf340cc34e00aa67e4588c61c32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.114188 4942 tablet_service.cc:1468] Processing CreateTablet for tablet 125f7265258e48368381aa54559eb510 (DEFAULT_TABLE table=default.Table_73a1fc5db8b944b3bddfc9cc2d03b002 [id=61973cf3444f4432be5e8a9fe2cae190]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.115227 4942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125f7265258e48368381aa54559eb510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.117208 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 125f7265258e48368381aa54559eb510 (DEFAULT_TABLE table=default.Table_73a1fc5db8b944b3bddfc9cc2d03b002 [id=61973cf3444f4432be5e8a9fe2cae190]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.117641 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7c352224d0a46c6a922bf0a0d765f17" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:39.118139 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125f7265258e48368381aa54559eb510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.118189 4975 raft_consensus.cc:3058] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:39.128402 4959 tablet_service.cc:1468] Processing CreateTablet for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (DEFAULT_TABLE table=default.Table_32a604ea7e0d4d9a8b477c5e6c1e87d5 [id=f588a91106544d889257bebe79a0a7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.129359 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f09f0381eb420ca57c18a5be6f2cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.131639 4975 raft_consensus.cc:2466] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:39.138734 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8122b26848141a39d2462dd3491f9cd" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:39.098479 5366 consensus_queue.cc:260] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.149288 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:39.150477 4394 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 94327cab01df4338aaec157736d106f6
I20250626 19:57:39.152282 4394 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34796 (request call id 104) took 2743 ms. Trace:
W20250626 19:57:39.152818 4755 consensus_peers.cc:487] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:39.155033 4399 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: default.Table_38472096f51645c8b48cf4841ee73622 [id=791d0e0df1e84922a9901aa6c456367b] alter complete (version 1)
I20250626 19:57:39.155872 4396 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 94cefa59de044a36a2afaaf5531128d8
W20250626 19:57:39.156082 4398 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0bcbb044d1549c88e5b83eeb5c709af consensus state has no leader
I20250626 19:57:39.156603 4398 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34796 (request call id 101) took 2917 ms. Trace:
I20250626 19:57:39.157159 4396 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34796 (request call id 110) took 1744 ms. Trace:
I20250626 19:57:39.159667 4397 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34796 (request call id 103) took 2759 ms. Trace:
I20250626 19:57:39.162406 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 94327cab01df4338aaec157736d106f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:36 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.162741 4956 tablet_service.cc:1515] Processing DeleteTablet for tablet 94cefa59de044a36a2afaaf5531128d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.163111 5653 tablet_replica.cc:331] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:39.163877 5653 raft_consensus.cc:2241] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:39.164358 5653 raft_consensus.cc:2270] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:39.164333 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 94cefa59de044a36a2afaaf5531128d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.166226 5653 ts_tablet_manager.cc:1905] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:39.157480 4396 rpcz_store.cc:276] 0626 19:57:37.412454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:37.412553 (+ 99us) service_pool.cc:225] Handling call
0626 19:57:37.413739 (+ 1186us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:37.414272 (+ 533us) catalog_manager.cc:2790] Modifying in-memory table state
0626 19:57:37.414389 (+ 117us) catalog_manager.cc:2799] Locking tablets
0626 19:57:37.427317 (+ 12928us) catalog_manager.cc:2813] Removing table and tablets from system table
0626 19:57:37.467591 (+ 40274us) catalog_manager.cc:2831] Removing table from by-name map
0626 19:57:37.467798 (+ 207us) catalog_manager.cc:2847] Committing in-memory state
0626 19:57:39.155752 (+1687954us) catalog_manager.cc:2851] Aborting table tasks
0626 19:57:39.157108 (+ 1356us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:37.429181 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:37.429537 (+ 356us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:37.429561 (+ 24us) write_op.cc:435] Acquired schema lock
0626 19:57:37.429573 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:37.429893 (+ 320us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:37.429935 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:37.429958 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:37.430205 (+ 247us) tablet.cc:699] Row locks acquired
0626 19:57:37.430219 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:57:37.430326 (+ 107us) write_op.cc:270] Start()
0626 19:57:37.430406 (+ 80us) write_op.cc:276] Timestamp: P: 1750967857430473 usec, L: 0
0626 19:57:37.430423 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:37.431016 (+ 593us) log.cc:844] Serialized 728 byte log entry
0626 19:57:37.465310 (+ 34294us) op_driver.cc:464] REPLICATION: finished
0626 19:57:37.466145 (+ 835us) write_op.cc:301] APPLY: starting
0626 19:57:37.466206 (+ 61us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:37.466304 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:37.466393 (+ 89us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:37.466602 (+ 209us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:37.466737 (+ 135us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:37.466804 (+ 67us) write_op.cc:312] APPLY: finished
0626 19:57:37.466991 (+ 187us) log.cc:844] Serialized 29 byte log entry
0626 19:57:37.467152 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:37.467220 (+ 68us) write_op.cc:454] Released schema lock
0626 19:57:37.467251 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:37.467468 (+ 217us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:37.467489 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":515,"apply.run_cpu_time_us":1291,"apply.run_wall_time_us":1290,"num_ops":2,"prepare.queue_time_us":93,"prepare.run_cpu_time_us":2028,"prepare.run_wall_time_us":2084,"raft.queue_time_us":2026,"raft.run_cpu_time_us":797,"raft.run_wall_time_us":793,"replication_time_us":34797}]]}
I20250626 19:57:39.152567 4394 rpcz_store.cc:276] 0626 19:57:36.409179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.951553 (+542374us) service_pool.cc:225] Handling call
0626 19:57:36.999372 (+ 47819us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:37.000865 (+ 1493us) catalog_manager.cc:2790] Modifying in-memory table state
0626 19:57:37.000983 (+ 118us) catalog_manager.cc:2799] Locking tablets
0626 19:57:37.001519 (+ 536us) catalog_manager.cc:2813] Removing table and tablets from system table
0626 19:57:37.081853 (+ 80334us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:37.081884 (+ 31us) write_op.cc:454] Released schema lock
0626 19:57:37.082245 (+ 361us) catalog_manager.cc:2831] Removing table from by-name map
0626 19:57:37.144344 (+ 62099us) catalog_manager.cc:2847] Committing in-memory state
0626 19:57:39.150388 (+2006044us) catalog_manager.cc:2851] Aborting table tasks
0626 19:57:39.152225 (+ 1837us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:37.062407 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:37.062812 (+ 405us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:37.062835 (+ 23us) write_op.cc:435] Acquired schema lock
0626 19:57:37.062846 (+ 11us) tablet.cc:660] Decoding operations
0626 19:57:37.063175 (+ 329us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:37.063212 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:37.063232 (+ 20us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:37.063485 (+ 253us) tablet.cc:699] Row locks acquired
0626 19:57:37.063497 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:37.063597 (+ 100us) write_op.cc:270] Start()
0626 19:57:37.063673 (+ 76us) write_op.cc:276] Timestamp: P: 1750967857063601 usec, L: 0
0626 19:57:37.063688 (+ 15us) op_driver.cc:348] REPLICATION: starting
0626 19:57:37.064183 (+ 495us) log.cc:844] Serialized 762 byte log entry
0626 19:57:37.077363 (+ 13180us) op_driver.cc:464] REPLICATION: finished
0626 19:57:37.077794 (+ 431us) write_op.cc:301] APPLY: starting
0626 19:57:37.077845 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:37.077951 (+ 106us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:37.078042 (+ 91us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:37.078258 (+ 216us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:37.078390 (+ 132us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:37.078433 (+ 43us) write_op.cc:312] APPLY: finished
0626 19:57:37.078593 (+ 160us) log.cc:844] Serialized 29 byte log entry
0626 19:57:37.078723 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:37.078812 (+ 89us) write_op.cc:454] Released schema lock
0626 19:57:37.078842 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":277,"apply.run_cpu_time_us":1236,"apply.run_wall_time_us":1236,"num_ops":2,"prepare.queue_time_us":53326,"prepare.run_cpu_time_us":2079,"prepare.run_wall_time_us":2078,"raft.queue_time_us":1756,"raft.run_cpu_time_us":666,"raft.run_wall_time_us":666,"replication_time_us":13586}]]}
I20250626 19:57:39.159977 4397 rpcz_store.cc:276] 0626 19:57:36.400345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.847117 (+446772us) service_pool.cc:225] Handling call
0626 19:57:37.004852 (+157735us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:37.005520 (+ 668us) catalog_manager.cc:3683] Apply alter schema
0626 19:57:37.008485 (+ 2965us) catalog_manager.cc:3715] Acquired catalog manager lock
0626 19:57:37.008674 (+ 189us) catalog_manager.cc:3873] Updating metadata on disk
0626 19:57:37.099665 (+ 90991us) catalog_manager.cc:3904] Committing alterations to in-memory state
0626 19:57:39.159605 (+2059940us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:37.064568 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:37.064915 (+ 347us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:37.064936 (+ 21us) write_op.cc:435] Acquired schema lock
0626 19:57:37.064948 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:37.065175 (+ 227us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:37.065208 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:37.065229 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:37.065360 (+ 131us) tablet.cc:699] Row locks acquired
0626 19:57:37.065372 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:37.065466 (+ 94us) write_op.cc:270] Start()
0626 19:57:37.065537 (+ 71us) write_op.cc:276] Timestamp: P: 1750967857065473 usec, L: 0
0626 19:57:37.065552 (+ 15us) op_driver.cc:348] REPLICATION: starting
0626 19:57:37.065998 (+ 446us) log.cc:844] Serialized 405 byte log entry
0626 19:57:37.094251 (+ 28253us) op_driver.cc:464] REPLICATION: finished
0626 19:57:37.096887 (+ 2636us) write_op.cc:301] APPLY: starting
0626 19:57:37.096959 (+ 72us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:37.097062 (+ 103us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:37.097165 (+ 103us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:37.097326 (+ 161us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:37.097470 (+ 144us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:37.097515 (+ 45us) write_op.cc:312] APPLY: finished
0626 19:57:37.097708 (+ 193us) log.cc:844] Serialized 23 byte log entry
0626 19:57:37.097918 (+ 210us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:37.097984 (+ 66us) write_op.cc:454] Released schema lock
0626 19:57:37.098017 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:37.098264 (+ 247us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:37.098294 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2469,"apply.run_cpu_time_us":1346,"apply.run_wall_time_us":1346,"mutex_wait_us":52732,"num_ops":1,"prepare.queue_time_us":54908,"prepare.run_cpu_time_us":1667,"prepare.run_wall_time_us":1666,"raft.queue_time_us":681,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":13,"replication_time_us":28599}]]}
I20250626 19:57:39.174176 4397 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_16e44e41e06c4d5f8e08d73dc5fa185e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:39.156869 4398 rpcz_store.cc:276] 0626 19:57:36.240100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.838881 (+598781us) service_pool.cc:225] Handling call
0626 19:57:36.997601 (+158720us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:36.998293 (+ 692us) catalog_manager.cc:3683] Apply alter schema
0626 19:57:36.998943 (+ 650us) catalog_manager.cc:3715] Acquired catalog manager lock
0626 19:57:36.999183 (+ 240us) catalog_manager.cc:3873] Updating metadata on disk
0626 19:57:37.105742 (+106559us) catalog_manager.cc:3904] Committing alterations to in-memory state
0626 19:57:39.156544 (+2050802us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:37.066336 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:37.066670 (+ 334us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:37.066691 (+ 21us) write_op.cc:435] Acquired schema lock
0626 19:57:37.066704 (+ 13us) tablet.cc:660] Decoding operations
0626 19:57:37.067714 (+ 1010us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:37.067752 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:37.067775 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:37.067901 (+ 126us) tablet.cc:699] Row locks acquired
0626 19:57:37.067918 (+ 17us) write_op.cc:260] PREPARE: finished
0626 19:57:37.068010 (+ 92us) write_op.cc:270] Start()
0626 19:57:37.068095 (+ 85us) write_op.cc:276] Timestamp: P: 1750967857068020 usec, L: 0
0626 19:57:37.068113 (+ 18us) op_driver.cc:348] REPLICATION: starting
0626 19:57:37.068530 (+ 417us) log.cc:844] Serialized 405 byte log entry
0626 19:57:37.094492 (+ 25962us) op_driver.cc:464] REPLICATION: finished
0626 19:57:37.098528 (+ 4036us) write_op.cc:301] APPLY: starting
0626 19:57:37.098582 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:37.098670 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:37.098796 (+ 126us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:37.098933 (+ 137us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:37.099062 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:37.099110 (+ 48us) write_op.cc:312] APPLY: finished
0626 19:57:37.099265 (+ 155us) log.cc:844] Serialized 23 byte log entry
0626 19:57:37.099394 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:37.099456 (+ 62us) write_op.cc:454] Released schema lock
0626 19:57:37.099487 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3887,"apply.run_cpu_time_us":1215,"apply.run_wall_time_us":7567,"mutex_wait_us":50942,"num_ops":1,"prepare.queue_time_us":54761,"prepare.run_cpu_time_us":1706,"prepare.run_wall_time_us":2568,"raft.queue_time_us":74,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":14,"replication_time_us":26303,"thread_start_us":20517,"threads_started":1}]]}
I20250626 19:57:39.178721 4398 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_f6018bf185744bb9981e34b1c75cc712"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:39.166646 4799 tablet_service.cc:1515] Processing DeleteTablet for tablet 94327cab01df4338aaec157736d106f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:36 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.181797 5438 tablet_replica.cc:331] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
W20250626 19:57:39.191989 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 49401.5us
I20250626 19:57:39.192844 5366 ts_tablet_manager.cc:1428] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.124s user 0.005s sys 0.000s
I20250626 19:57:39.193744 5366 tablet_bootstrap.cc:492] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:39.193629 4756 leader_election.cc:304] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:39.195436 5429 consensus_queue.cc:1035] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:39.196975 5429 raft_consensus.cc:2802] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:39.202971 5429 raft_consensus.cc:695] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:39.196651 5686 tablet.cc:1722] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250626 19:57:39.203801 5429 consensus_queue.cc:237] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:39.176482 4397 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16e44e41e06c4d5f8e08d73dc5fa185e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:39.216415 5653 ts_tablet_manager.cc:1918] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:39.216965 5653 log.cc:1199] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/94327cab01df4338aaec157736d106f6
I20250626 19:57:39.218591 5653 ts_tablet_manager.cc:1939] T 94327cab01df4338aaec157736d106f6 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:39.231206 5653 tablet_replica.cc:331] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:39.232215 5653 raft_consensus.cc:2241] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:39.237299 5653 raft_consensus.cc:2270] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:39.239315 5653 ts_tablet_manager.cc:1905] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:39.240684 5353 tablet_bootstrap.cc:492] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:39.241175 5353 ts_tablet_manager.cc:1397] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.233s user 0.020s sys 0.030s
I20250626 19:57:39.243741 5353 raft_consensus.cc:357] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.244381 5353 raft_consensus.cc:383] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:39.244638 5353 raft_consensus.cc:738] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:39.210729 5366 tablet_bootstrap.cc:654] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:39.089876 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82823fd9ceb1485386616319fa7aeb43" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:39.248994 4670 raft_consensus.cc:3058] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:39.248375 4390 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: default.Table_c0263194234444158f569a2abbfa91b5 [id=089059dab32a41179048a6eb4473a99a] alter complete (version 2)
I20250626 19:57:39.095391 5410 ts_tablet_manager.cc:1428] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.068s user 0.002s sys 0.004s
I20250626 19:57:39.255719 5410 tablet_bootstrap.cc:492] T 419c4018256546ef9431ac2db619ee33 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:39.108769 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7c352224d0a46c6a922bf0a0d765f17" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:39.111685 4652 tablet_service.cc:1468] Processing CreateTablet for tablet ef1abf340cc34e00aa67e4588c61c32d (DEFAULT_TABLE table=default.Table_e4d8a4cf39764c5fa002644d60f964b3 [id=d4284cd20f084362a5e9c533f5909b26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.260047 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1abf340cc34e00aa67e4588c61c32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.123973 4651 tablet_service.cc:1468] Processing CreateTablet for tablet 125f7265258e48368381aa54559eb510 (DEFAULT_TABLE table=default.Table_73a1fc5db8b944b3bddfc9cc2d03b002 [id=61973cf3444f4432be5e8a9fe2cae190]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.263476 4651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125f7265258e48368381aa54559eb510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.273707 5438 raft_consensus.cc:2241] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:39.127612 4649 tablet_service.cc:1468] Processing CreateTablet for tablet d8be634cb36d40dca3f9338dab93ad2c (DEFAULT_TABLE table=default.Table_6dc13f5ceb674e40919de2ca36d9427a [id=160181772a0a40ebac8a3fe697667b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.137421 4671 raft_consensus.cc:3058] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:39.274873 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8be634cb36d40dca3f9338dab93ad2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.155017 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet 94327cab01df4338aaec157736d106f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:36 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:39.160670 4648 tablet_service.cc:1515] Processing DeleteTablet for tablet 94cefa59de044a36a2afaaf5531128d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:39.282821 5624 raft_consensus.cc:491] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:39.224020 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 94327cab01df4338aaec157736d106f6 (table default.Table_beec4e8c60ae4e6880c0ff709d3c0ff3 [id=6e84598eef744e8c9b1d918036bc67ec]) successfully deleted
I20250626 19:57:39.283265 5624 raft_consensus.cc:513] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:39.283571 4385 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 59607.3us
I20250626 19:57:39.285753 5664 consensus_queue.cc:1035] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:57:39.288098 4398 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6018bf185744bb9981e34b1c75cc712 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:39.296625 5438 raft_consensus.cc:2270] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:39.296339 5353 consensus_queue.cc:260] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.298357 5438 ts_tablet_manager.cc:1905] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:39.282316 4755 leader_election.cc:343] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:39.299433 5653 ts_tablet_manager.cc:1918] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:39.299912 5653 log.cc:1199] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/94cefa59de044a36a2afaaf5531128d8
I20250626 19:57:39.299839 5624 leader_election.cc:290] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:39.300626 5653 ts_tablet_manager.cc:1939] T 94cefa59de044a36a2afaaf5531128d8 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:39.303079 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e60b4d53c40c4dffab928f0e3fb4a80b" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:39.306308 4670 raft_consensus.cc:2466] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:39.306705 4795 tablet_service.cc:1468] Processing CreateTablet for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (DEFAULT_TABLE table=default.Table_32a604ea7e0d4d9a8b477c5e6c1e87d5 [id=f588a91106544d889257bebe79a0a7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.307938 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f09f0381eb420ca57c18a5be6f2cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.292380 5410 tablet_bootstrap.cc:654] T 419c4018256546ef9431ac2db619ee33 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:39.327422 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 94cefa59de044a36a2afaaf5531128d8 (table default.Table_c580e8a6ee5d417caa2f7d6811a68349 [id=b8bf68764b2c4a94b7299fa273983e06]) successfully deleted
I20250626 19:57:39.342267 5410 tablet_bootstrap.cc:492] T 419c4018256546ef9431ac2db619ee33 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:39.342924 5410 ts_tablet_manager.cc:1397] T 419c4018256546ef9431ac2db619ee33 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.087s user 0.009s sys 0.001s
I20250626 19:57:39.343559 4950 tablet_service.cc:1468] Processing CreateTablet for tablet 83f0431638144710b598b3f87d9a576e (DEFAULT_TABLE table=default.Table_e298729c4c744ce3834f9f02a49d4349 [id=62e85ab3ba61489d992fc52e0e1d9a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.344674 4950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83f0431638144710b598b3f87d9a576e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.354130 4397 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 121) took 1329 ms. Trace:
I20250626 19:57:39.358857 4800 tablet_service.cc:1515] Processing DeleteTablet for tablet 1cb6bda529624ac5b5a87a03b4a02d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 465ed0fe56fb4960a58e92b9b3d594e6) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.352036 5353 ts_tablet_manager.cc:1428] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.110s user 0.006s sys 0.000s
I20250626 19:57:39.361312 5353 tablet_bootstrap.cc:492] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:39.354538 4397 rpcz_store.cc:276] 0626 19:57:38.024692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.173793 (+1149101us) service_pool.cc:225] Handling call
0626 19:57:39.211294 (+ 37501us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:39.213028 (+ 1734us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:39.314136 (+101108us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:39.353794 (+ 39658us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:39.354066 (+ 272us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:39.215212 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:39.215571 (+ 359us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:39.215592 (+ 21us) write_op.cc:435] Acquired schema lock
0626 19:57:39.215606 (+ 14us) tablet.cc:660] Decoding operations
0626 19:57:39.215859 (+ 253us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:39.215897 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:39.215921 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:39.216117 (+ 196us) tablet.cc:699] Row locks acquired
0626 19:57:39.216131 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:57:39.216223 (+ 92us) write_op.cc:270] Start()
0626 19:57:39.216301 (+ 78us) write_op.cc:276] Timestamp: P: 1750967859216274 usec, L: 0
0626 19:57:39.216318 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:39.231074 (+ 14756us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b5400020920. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72b481 00007fe78f73b9f0 00007fe78f7ad722 00007fe78f7a2b57 00007fe78f7a2994 00007fe78fc0f811 00007fe78fc0efe8 00007fe78fc13c81 00007fe78fc13c39 00007fe78fc13bc9 00007fe78fc13b91
0626 19:57:39.231597 (+ 523us) log.cc:844] Serialized 433 byte log entry
0626 19:57:39.235403 (+ 3806us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b5400020920. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72b481 00007fe78f73760f 00007fe78f720372 00007fe78f725781 00007fe78f725709 00007fe78f725699 00007fe78f725661 00007fe78f72482d 000000000059bcd4 000000000059bb09 00007fe78885b564
0626 19:57:39.309985 (+ 74582us) op_driver.cc:464] REPLICATION: finished
0626 19:57:39.310240 (+ 255us) write_op.cc:301] APPLY: starting
0626 19:57:39.310293 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:39.310404 (+ 111us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:39.310430 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:39.310899 (+ 469us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:39.311053 (+ 154us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:39.311100 (+ 47us) write_op.cc:312] APPLY: finished
0626 19:57:39.311284 (+ 184us) log.cc:844] Serialized 29 byte log entry
0626 19:57:39.311429 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:39.311499 (+ 70us) write_op.cc:454] Released schema lock
0626 19:57:39.311533 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":97,"apply.run_cpu_time_us":1477,"apply.run_wall_time_us":1476,"num_ops":2,"prepare.queue_time_us":1067,"prepare.run_cpu_time_us":3158,"prepare.run_wall_time_us":16688,"raft.queue_time_us":4077,"raft.run_cpu_time_us":1836,"raft.run_wall_time_us":3695,"replication_time_us":93570,"spinlock_wait_cycles":31758336}]]}
W20250626 19:57:39.359691 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 167110us
I20250626 19:57:39.368058 4756 leader_election.cc:304] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
W20250626 19:57:39.369328 4756 leader_election.cc:343] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:39.371577 4390 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 465ed0fe56fb4960a58e92b9b3d594e6 consensus state has no leader
I20250626 19:57:39.360556 4796 tablet_service.cc:1515] Processing DeleteTablet for tablet a31f1a7739544706bff2a500d6026086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:34 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.373698 4756 leader_election.cc:304] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:39.375272 5366 tablet_bootstrap.cc:492] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:39.375726 5366 ts_tablet_manager.cc:1397] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.182s user 0.010s sys 0.000s
I20250626 19:57:39.372948 5642 raft_consensus.cc:2802] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:39.376317 5506 raft_consensus.cc:2747] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:39.377027 5642 raft_consensus.cc:695] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:39.377890 5642 consensus_queue.cc:237] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.387391 4796 tablet_service.cc:1515] Processing DeleteTablet for tablet 82823fd9ceb1485386616319fa7aeb43 with delete_type TABLET_DATA_DELETED (replaced by tablet f6ec952321924098a142f1438a283aab) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.389129 4799 tablet_service.cc:1515] Processing DeleteTablet for tablet c92deaa237064dee8f866e3a7a355ca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:34 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.389811 4796 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c03691d021e4ee19d5bb287ab633a3c with delete_type TABLET_DATA_DELETED (replaced by tablet de0724ba71824a5daaf6f3708b4431b7) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.390055 4799 tablet_service.cc:1515] Processing DeleteTablet for tablet 36997dab0ab3430bba63a90805759b30 with delete_type TABLET_DATA_DELETED (replaced by tablet fafbf98e7f4544b6ad96c1c5ab6c82bc) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.391149 4799 tablet_service.cc:1515] Processing DeleteTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.392108 4799 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ab0320af47049cca3ca87172b9fe5e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 609d36d46bb84017b2a3a7ad48bfccfc) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.393920 4800 tablet_service.cc:1515] Processing DeleteTablet for tablet 7391b9eaea3e426ba4118fc8b53b72a6 with delete_type TABLET_DATA_DELETED (replaced by tablet d8399c219d0e4485bb7926eefd78f210) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.396226 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 83f0431638144710b598b3f87d9a576e (DEFAULT_TABLE table=default.Table_e298729c4c744ce3834f9f02a49d4349 [id=62e85ab3ba61489d992fc52e0e1d9a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.401146 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83f0431638144710b598b3f87d9a576e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.406298 5366 raft_consensus.cc:357] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.406953 5366 raft_consensus.cc:383] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:39.407215 5366 raft_consensus.cc:738] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:39.407859 5366 consensus_queue.cc:260] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.409775 4800 tablet_service.cc:1515] Processing DeleteTablet for tablet af58f34e2248432fae5c084bd3388028 with delete_type TABLET_DATA_DELETED (replaced by tablet e60b4d53c40c4dffab928f0e3fb4a80b) from {username='slave'} at 127.0.0.1:40742
W20250626 19:57:39.417953 4611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 221825us
I20250626 19:57:39.419657 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e60b4d53c40c4dffab928f0e3fb4a80b" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:39.421847 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 1cb6bda529624ac5b5a87a03b4a02d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 465ed0fe56fb4960a58e92b9b3d594e6) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:39.421779 4648 tablet_service.cc:1468] Processing CreateTablet for tablet 83f0431638144710b598b3f87d9a576e (DEFAULT_TABLE table=default.Table_e298729c4c744ce3834f9f02a49d4349 [id=62e85ab3ba61489d992fc52e0e1d9a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.422788 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ab0320af47049cca3ca87172b9fe5e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 609d36d46bb84017b2a3a7ad48bfccfc) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:39.423588 4652 tablet_service.cc:1468] Processing CreateTablet for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (DEFAULT_TABLE table=default.Table_32a604ea7e0d4d9a8b477c5e6c1e87d5 [id=f588a91106544d889257bebe79a0a7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.410038 4796 tablet_service.cc:1515] Processing DeleteTablet for tablet 499f6ae3262940b4859c14a53722482f with delete_type TABLET_DATA_DELETED (replaced by tablet 6b8473259485460783f62f8193d5b02c) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.410305 4793 tablet_service.cc:1515] Processing DeleteTablet for tablet 5785508e37624f02bc2f6cc93007e3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:28 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.410591 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet fd6543699e254400b3516aece4486fe0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1474bfa74a6346bf9e3ff49019f6c9b3) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.426657 4794 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b3bed19e7e34d24b995840a3bde5ef0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c113e03594b428a979435618fe373c6) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.427503 4797 tablet_service.cc:1468] Processing CreateTablet for tablet d7c5aba053ca4535ad2376b2066cfe47 (DEFAULT_TABLE table=default.Table_7a151cd2d08a4ec9aca048d41f8a5c04 [id=0245cc6d105e42388545608e0a267fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.430733 4797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c5aba053ca4535ad2376b2066cfe47. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:39.430845 4390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 63) took 1955 ms (client timeout 1000 ms). Trace:
I20250626 19:57:39.385445 4397 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_151fcc3e95fc46718ce92d0cfc4dcbaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:39.432264 4397 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_151fcc3e95fc46718ce92d0cfc4dcbaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:39.432219 5410 raft_consensus.cc:357] T 419c4018256546ef9431ac2db619ee33 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.432850 5410 raft_consensus.cc:383] T 419c4018256546ef9431ac2db619ee33 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:39.433099 5410 raft_consensus.cc:738] T 419c4018256546ef9431ac2db619ee33 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:39.433717 5410 consensus_queue.cc:260] T 419c4018256546ef9431ac2db619ee33 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.434917 5366 ts_tablet_manager.cc:1428] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.052s user 0.003s sys 0.002s
I20250626 19:57:39.435799 5366 tablet_bootstrap.cc:492] T 6cc7a336385f499ca19566d5f2901e98 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:39.444553 5366 tablet_bootstrap.cc:654] T 6cc7a336385f499ca19566d5f2901e98 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:39.452436 5664 raft_consensus.cc:491] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:39.452911 5664 raft_consensus.cc:513] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.454661 5664 leader_election.cc:290] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:39.458634 4395 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 77) took 2667 ms (client timeout 1000 ms). Trace:
I20250626 19:57:39.459717 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "211f6137d9b84afa9aee6432440f7eb8" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
W20250626 19:57:39.458990 4395 rpcz_store.cc:269] 0626 19:57:36.790732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.825478 (+ 34746us) service_pool.cc:225] Handling call
0626 19:57:39.353723 (+2528245us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b6800133160. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe791479af1 00007fe79145712a 00007fe791455868 00007fe791453e90 00007fe7915560ae 00007fe78f26c624 00007fe78f26c5b2 00007fe78f26c4e1 00007fe78f26c45c 00007fe78f26b712 00007fe78d56fdcc
0626 19:57:39.458569 (+104846us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:37.884712 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:37.885067 (+ 355us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:37.885094 (+ 27us) write_op.cc:435] Acquired schema lock
0626 19:57:37.885106 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:37.885346 (+ 240us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:37.885384 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:37.885406 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:37.885577 (+ 171us) tablet.cc:699] Row locks acquired
0626 19:57:37.885592 (+ 15us) write_op.cc:260] PREPARE: finished
0626 19:57:37.885689 (+ 97us) write_op.cc:270] Start()
0626 19:57:37.885770 (+ 81us) write_op.cc:276] Timestamp: P: 1750967857885822 usec, L: 0
0626 19:57:37.885785 (+ 15us) op_driver.cc:348] REPLICATION: starting
0626 19:57:37.900737 (+ 14952us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b5400020998. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe7904dc5d4 00007fe78f779bf9 00007fe78f735b51 00007fe78f73582a 00007fe78f7a2555 00007fe78f7a29f9 00007fe78fc0f811 00007fe78fc0efe8 00007fe78fc13c81 00007fe78fc13c39 00007fe78fc13bc9
0626 19:57:37.901006 (+ 269us) log.cc:844] Serialized 530 byte log entry
0626 19:57:37.909825 (+ 8819us) spinlock_profiling.cc:243] Waited 6.14 ms on lock 0x7b5800085ab0. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7206fc 00007fe78f725781 00007fe78f725709 00007fe78f725699 00007fe78f725661 00007fe78f72482d 000000000059bcd4 000000000059bb09 00007fe78885b564 00007fe78885d8b1
0626 19:57:37.913307 (+ 3482us) op_driver.cc:464] REPLICATION: finished
0626 19:57:37.913624 (+ 317us) write_op.cc:301] APPLY: starting
0626 19:57:37.913671 (+ 47us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:37.913782 (+ 111us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:37.913810 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:37.914028 (+ 218us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:37.914214 (+ 186us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:37.914278 (+ 64us) write_op.cc:312] APPLY: finished
0626 19:57:37.915557 (+ 1279us) log.cc:844] Serialized 23 byte log entry
0626 19:57:37.915713 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:37.915775 (+ 62us) write_op.cc:454] Released schema lock
0626 19:57:37.915809 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:37.916776 (+ 967us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:37.916801 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":60426,"reported_tablets":46,"spinlock_wait_cycles":25801088,"child_traces":[["op",{"apply.queue_time_us":174,"apply.run_cpu_time_us":1359,"apply.run_wall_time_us":3108,"num_ops":1,"prepare.queue_time_us":95,"prepare.run_cpu_time_us":3283,"prepare.run_wall_time_us":16603,"raft.queue_time_us":1163,"raft.run_cpu_time_us":2451,"raft.run_wall_time_us":8319,"replication_time_us":27428,"spinlock_wait_cycles":45204096}]]}
I20250626 19:57:39.470659 4397 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 123) took 1406 ms. Trace:
I20250626 19:57:39.472353 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b3bed19e7e34d24b995840a3bde5ef0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c113e03594b428a979435618fe373c6) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:39.392901 4398 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 122) took 1368 ms. Trace:
W20250626 19:57:39.399132 4385 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e0bcbb044d1549c88e5b83eeb5c709af consensus state has no leader
I20250626 19:57:39.484592 5410 ts_tablet_manager.cc:1428] T 419c4018256546ef9431ac2db619ee33 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.141s user 0.003s sys 0.002s
W20250626 19:57:39.391821 4391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 72) took 1841 ms (client timeout 1000 ms). Trace:
W20250626 19:57:39.486996 4911 leader_election.cc:343] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:39.487326 4942 tablet_service.cc:1515] Processing DeleteTablet for tablet 7391b9eaea3e426ba4118fc8b53b72a6 with delete_type TABLET_DATA_DELETED (replaced by tablet d8399c219d0e4485bb7926eefd78f210) from {username='slave'} at 127.0.0.1:41356
W20250626 19:57:39.485003 4391 rpcz_store.cc:269] 0626 19:57:37.550009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:37.561969 (+ 11960us) service_pool.cc:225] Handling call
0626 19:57:38.573734 (+1011765us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:38.573757 (+ 23us) write_op.cc:454] Released schema lock
0626 19:57:39.352086 (+778329us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b6800133160. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe791479af1 00007fe79145712a 00007fe791455868 00007fe791453e90 00007fe7915560ae 00007fe78f26c624 00007fe78f26c5b2 00007fe78f26c4e1 00007fe78f26c45c 00007fe78f26b712 00007fe78d56fdcc
0626 19:57:39.391755 (+ 39669us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:38.558961 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:38.559335 (+ 374us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:38.559369 (+ 34us) write_op.cc:435] Acquired schema lock
0626 19:57:38.559388 (+ 19us) tablet.cc:660] Decoding operations
0626 19:57:38.559762 (+ 374us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:38.559816 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:38.559849 (+ 33us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:38.560110 (+ 261us) tablet.cc:699] Row locks acquired
0626 19:57:38.560124 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:57:38.560228 (+ 104us) write_op.cc:270] Start()
0626 19:57:38.560305 (+ 77us) write_op.cc:276] Timestamp: P: 1750967858560227 usec, L: 0
0626 19:57:38.560318 (+ 13us) op_driver.cc:348] REPLICATION: starting
0626 19:57:38.560806 (+ 488us) log.cc:844] Serialized 907 byte log entry
0626 19:57:38.567886 (+ 7080us) op_driver.cc:464] REPLICATION: finished
0626 19:57:38.568486 (+ 600us) write_op.cc:301] APPLY: starting
0626 19:57:38.568537 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:38.568639 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:38.568718 (+ 79us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:38.568960 (+ 242us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:38.569088 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:38.569128 (+ 40us) write_op.cc:312] APPLY: finished
0626 19:57:38.569303 (+ 175us) log.cc:844] Serialized 29 byte log entry
0626 19:57:38.569457 (+ 154us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:38.569540 (+ 83us) write_op.cc:454] Released schema lock
0626 19:57:38.569570 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":86998,"reported_tablets":51,"spinlock_wait_cycles":61575936,"child_traces":[["op",{"apply.queue_time_us":450,"apply.run_cpu_time_us":1246,"apply.run_wall_time_us":1244,"num_ops":2,"prepare.queue_time_us":8611,"prepare.run_cpu_time_us":2146,"prepare.run_wall_time_us":2145,"raft.queue_time_us":1042,"raft.run_cpu_time_us":607,"raft.run_wall_time_us":608,"replication_time_us":7486}]]}
I20250626 19:57:39.488265 4391 master_service.cc:948] ReplaceTablet: received request to replace tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:39.488618 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 36997dab0ab3430bba63a90805759b30 with delete_type TABLET_DATA_DELETED (replaced by tablet fafbf98e7f4544b6ad96c1c5ab6c82bc) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.489259 5653 tablet_replica.cc:331] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:39.490068 5653 raft_consensus.cc:2241] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:39.490145 4956 tablet_service.cc:1515] Processing DeleteTablet for tablet 82823fd9ceb1485386616319fa7aeb43 with delete_type TABLET_DATA_DELETED (replaced by tablet f6ec952321924098a142f1438a283aab) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.490509 5653 raft_consensus.cc:2270] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:39.492270 4952 tablet_service.cc:1468] Processing CreateTablet for tablet d7c5aba053ca4535ad2376b2066cfe47 (DEFAULT_TABLE table=default.Table_7a151cd2d08a4ec9aca048d41f8a5c04 [id=0245cc6d105e42388545608e0a267fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.492681 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ab0320af47049cca3ca87172b9fe5e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 609d36d46bb84017b2a3a7ad48bfccfc) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.493459 4952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c5aba053ca4535ad2376b2066cfe47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.494930 5653 ts_tablet_manager.cc:1905] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:39.495443 4957 tablet_service.cc:1468] Processing CreateTablet for tablet e94a73b50b564a97bbee0f5015bb8805 (DEFAULT_TABLE table=default.Table_fbcc45f0e38f4ed184900c0f58067165 [id=10f3db9788ba4edda2aab300e7313abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.496317 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e94a73b50b564a97bbee0f5015bb8805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.497772 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "211f6137d9b84afa9aee6432440f7eb8" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:39.499418 4951 tablet_service.cc:1515] Processing DeleteTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.500538 4942 tablet_service.cc:1515] Processing DeleteTablet for tablet 94cefa59de044a36a2afaaf5531128d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:41356
W20250626 19:57:39.501662 4912 leader_election.cc:343] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:39.502039 4912 leader_election.cc:304] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:39.503011 5664 raft_consensus.cc:2747] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:39.504642 5674 tablet.cc:1722] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250626 19:57:39.507093 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 7391b9eaea3e426ba4118fc8b53b72a6 because the tablet was not found. No further retry: Not found: Tablet not found: 7391b9eaea3e426ba4118fc8b53b72a6
W20250626 19:57:39.507951 4385 catalog_manager.cc:5001] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): alter failed for tablet 3c03691d021e4ee19d5bb287ab633a3c (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]),no further retry: Not found: Tablet not found: 3c03691d021e4ee19d5bb287ab633a3c
I20250626 19:57:39.511013 5686 tablet.cc:1722] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:39.515059 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet 94327cab01df4338aaec157736d106f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:36 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.520798 4395 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_c8720865c8c742bc941a7d2665c618f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:39.522575 4395 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8720865c8c742bc941a7d2665c618f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:39.520227 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet af58f34e2248432fae5c084bd3388028 with delete_type TABLET_DATA_DELETED (replaced by tablet e60b4d53c40c4dffab928f0e3fb4a80b) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.525890 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a8f494f8614e7ebb8ca880b6c80e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:27 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.527397 4648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83f0431638144710b598b3f87d9a576e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.528957 4651 tablet_service.cc:1468] Processing CreateTablet for tablet d7c5aba053ca4535ad2376b2066cfe47 (DEFAULT_TABLE table=default.Table_7a151cd2d08a4ec9aca048d41f8a5c04 [id=0245cc6d105e42388545608e0a267fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.530033 4651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c5aba053ca4535ad2376b2066cfe47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.530872 5410 tablet_bootstrap.cc:492] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:39.515975 4398 rpcz_store.cc:276] 0626 19:57:38.024918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.178330 (+1153412us) service_pool.cc:225] Handling call
0626 19:57:39.352976 (+174646us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:39.363899 (+ 10923us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:39.392155 (+ 28256us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:39.392180 (+ 25us) write_op.cc:454] Released schema lock
0626 19:57:39.392374 (+ 194us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:39.392629 (+ 255us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:39.392853 (+ 224us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:39.365207 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:39.365570 (+ 363us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:39.365596 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:39.365610 (+ 14us) tablet.cc:660] Decoding operations
0626 19:57:39.365866 (+ 256us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:39.365906 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:39.365930 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:39.366164 (+ 234us) tablet.cc:699] Row locks acquired
0626 19:57:39.366178 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:57:39.366282 (+ 104us) write_op.cc:270] Start()
0626 19:57:39.366360 (+ 78us) write_op.cc:276] Timestamp: P: 1750967859366388 usec, L: 0
0626 19:57:39.366377 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:39.366990 (+ 613us) log.cc:844] Serialized 433 byte log entry
0626 19:57:39.374361 (+ 7371us) op_driver.cc:464] REPLICATION: finished
0626 19:57:39.374607 (+ 246us) write_op.cc:301] APPLY: starting
0626 19:57:39.374660 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:39.374787 (+ 127us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:39.374874 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:39.375276 (+ 402us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:39.375407 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:39.375458 (+ 51us) write_op.cc:312] APPLY: finished
0626 19:57:39.375632 (+ 174us) log.cc:844] Serialized 29 byte log entry
0626 19:57:39.375771 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:39.375839 (+ 68us) write_op.cc:454] Released schema lock
0626 19:57:39.375875 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":106,"apply.run_cpu_time_us":1461,"apply.run_wall_time_us":1456,"num_ops":2,"prepare.queue_time_us":129,"prepare.run_cpu_time_us":2023,"prepare.run_wall_time_us":19003,"raft.queue_time_us":890,"raft.run_cpu_time_us":592,"raft.run_wall_time_us":593,"replication_time_us":7897}]]}
I20250626 19:57:39.532023 4398 master_service.cc:948] ReplaceTablet: received request to replace tablet f43aed8f07844c5ca727ae6df63896df from {username='slave'} at 127.0.0.1:34796
W20250626 19:57:39.479626 4399 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 71) took 3016 ms (client timeout 1000 ms). Trace:
I20250626 19:57:39.541013 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet 499f6ae3262940b4859c14a53722482f with delete_type TABLET_DATA_DELETED (replaced by tablet 6b8473259485460783f62f8193d5b02c) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.546713 4942 tablet_service.cc:1515] Processing DeleteTablet for tablet 1cb6bda529624ac5b5a87a03b4a02d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 465ed0fe56fb4960a58e92b9b3d594e6) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.547041 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b3bed19e7e34d24b995840a3bde5ef0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c113e03594b428a979435618fe373c6) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.553179 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f09f0381eb420ca57c18a5be6f2cc2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:39.431142 4390 rpcz_store.cc:269] 0626 19:57:37.475319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:37.475447 (+ 128us) service_pool.cc:225] Handling call
0626 19:57:38.344305 (+868858us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:38.344331 (+ 26us) write_op.cc:454] Released schema lock
0626 19:57:39.347606 (+1003275us) spinlock_profiling.cc:243] Waited 40 ms on lock 0x7b6800133160. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe791479af1 00007fe79145712a 00007fe791455868 00007fe791453e90 00007fe7915560ae 00007fe78f26c624 00007fe78f26c5b2 00007fe78f26c4e1 00007fe78f26c45c 00007fe78f26b712 00007fe78d56fdcc
0626 19:57:39.430775 (+ 83169us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:38.324423 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:38.324789 (+ 366us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:38.324812 (+ 23us) write_op.cc:435] Acquired schema lock
0626 19:57:38.324824 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:38.325118 (+ 294us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:38.325159 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:38.325182 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:38.325513 (+ 331us) tablet.cc:699] Row locks acquired
0626 19:57:38.325536 (+ 23us) write_op.cc:260] PREPARE: finished
0626 19:57:38.325654 (+ 118us) write_op.cc:270] Start()
0626 19:57:38.325739 (+ 85us) write_op.cc:276] Timestamp: P: 1750967858325751 usec, L: 0
0626 19:57:38.325755 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:57:38.326244 (+ 489us) log.cc:844] Serialized 907 byte log entry
0626 19:57:38.337015 (+ 10771us) op_driver.cc:464] REPLICATION: finished
0626 19:57:38.337407 (+ 392us) write_op.cc:301] APPLY: starting
0626 19:57:38.337458 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:38.337555 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:38.337639 (+ 84us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:38.337862 (+ 223us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:38.338000 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:38.338040 (+ 40us) write_op.cc:312] APPLY: finished
0626 19:57:38.338217 (+ 177us) log.cc:844] Serialized 29 byte log entry
0626 19:57:38.338360 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:38.338429 (+ 69us) write_op.cc:454] Released schema lock
0626 19:57:38.338461 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0626 19:57:39.213263 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:39.213620 (+ 357us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:39.213644 (+ 24us) write_op.cc:435] Acquired schema lock
0626 19:57:39.213657 (+ 13us) tablet.cc:660] Decoding operations
0626 19:57:39.213891 (+ 234us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:39.213930 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:39.213951 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:39.214087 (+ 136us) tablet.cc:699] Row locks acquired
0626 19:57:39.214101 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:57:39.214205 (+ 104us) write_op.cc:270] Start()
0626 19:57:39.214288 (+ 83us) write_op.cc:276] Timestamp: P: 1750967859214250 usec, L: 0
0626 19:57:39.214309 (+ 21us) op_driver.cc:348] REPLICATION: starting
0626 19:57:39.214801 (+ 492us) log.cc:844] Serialized 356 byte log entry
0626 19:57:39.231929 (+ 17128us) op_driver.cc:464] REPLICATION: finished
0626 19:57:39.232687 (+ 758us) write_op.cc:301] APPLY: starting
0626 19:57:39.232747 (+ 60us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:39.232846 (+ 99us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:39.232949 (+ 103us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:39.233125 (+ 176us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:39.233275 (+ 150us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:39.233327 (+ 52us) write_op.cc:312] APPLY: finished
0626 19:57:39.233505 (+ 178us) log.cc:844] Serialized 23 byte log entry
0626 19:57:39.233660 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:39.233726 (+ 66us) write_op.cc:454] Released schema lock
0626 19:57:39.233759 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:39.233999 (+ 240us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:39.234027 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":18353,"reported_tablets":47,"spinlock_wait_cycles":91907456,"child_traces":[["op",{"apply.queue_time_us":255,"apply.run_cpu_time_us":1229,"apply.run_wall_time_us":1227,"num_ops":2,"prepare.queue_time_us":12364,"prepare.run_cpu_time_us":2165,"prepare.run_wall_time_us":2164,"raft.queue_time_us":1534,"raft.run_cpu_time_us":638,"raft.run_wall_time_us":636,"replication_time_us":11163}],["op",{"apply.queue_time_us":589,"apply.run_cpu_time_us":1283,"apply.run_wall_time_us":1282,"num_ops":1,"prepare.queue_time_us":106,"prepare.run_cpu_time_us":1838,"prepare.run_wall_time_us":1838,"raft.queue_time_us":3762,"raft.run_cpu_time_us":666,"raft.run_wall_time_us":666,"replication_time_us":17508}]]}
I20250626 19:57:39.526796 4397 rpcz_store.cc:276] 0626 19:57:38.064444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.362686 (+1298242us) service_pool.cc:225] Handling call
0626 19:57:39.432629 (+ 69943us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:39.434093 (+ 1464us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:39.469748 (+ 35655us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:39.469778 (+ 30us) write_op.cc:454] Released schema lock
0626 19:57:39.470019 (+ 241us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:39.470323 (+ 304us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:39.470603 (+ 280us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:39.435601 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:39.435965 (+ 364us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:39.436000 (+ 35us) write_op.cc:435] Acquired schema lock
0626 19:57:39.436014 (+ 14us) tablet.cc:660] Decoding operations
0626 19:57:39.436293 (+ 279us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:39.436335 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:39.436357 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:39.436543 (+ 186us) tablet.cc:699] Row locks acquired
0626 19:57:39.436558 (+ 15us) write_op.cc:260] PREPARE: finished
0626 19:57:39.436663 (+ 105us) write_op.cc:270] Start()
0626 19:57:39.436745 (+ 82us) write_op.cc:276] Timestamp: P: 1750967859436797 usec, L: 0
0626 19:57:39.436762 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:39.437233 (+ 471us) log.cc:844] Serialized 433 byte log entry
0626 19:57:39.444447 (+ 7214us) op_driver.cc:464] REPLICATION: finished
0626 19:57:39.444788 (+ 341us) write_op.cc:301] APPLY: starting
0626 19:57:39.444841 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:39.444938 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:39.445050 (+ 112us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:39.445496 (+ 446us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:39.445651 (+ 155us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:39.445696 (+ 45us) write_op.cc:312] APPLY: finished
0626 19:57:39.445876 (+ 180us) log.cc:844] Serialized 29 byte log entry
0626 19:57:39.446027 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:39.446113 (+ 86us) write_op.cc:454] Released schema lock
0626 19:57:39.446145 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":165,"apply.run_cpu_time_us":1533,"apply.run_wall_time_us":1532,"num_ops":2,"prepare.queue_time_us":88,"prepare.run_cpu_time_us":1984,"prepare.run_wall_time_us":7393,"raft.queue_time_us":728,"raft.run_cpu_time_us":704,"raft.run_wall_time_us":700,"replication_time_us":7587}]]}
I20250626 19:57:39.560009 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet a31f1a7739544706bff2a500d6026086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:34 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.560782 5653 ts_tablet_manager.cc:1918] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:39.562742 4397 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_d23ee99af7b845b78d2200b962f681a6" } modify_external_catalogs: true
I20250626 19:57:39.563807 4397 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_d23ee99af7b845b78d2200b962f681a6" } modify_external_catalogs: true
I20250626 19:57:39.572510 4656 tablet_service.cc:1468] Processing CreateTablet for tablet e94a73b50b564a97bbee0f5015bb8805 (DEFAULT_TABLE table=default.Table_fbcc45f0e38f4ed184900c0f58067165 [id=10f3db9788ba4edda2aab300e7313abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.573658 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e94a73b50b564a97bbee0f5015bb8805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.558657 5438 ts_tablet_manager.cc:1918] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:39.579250 5438 log.cc:1199] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/94cefa59de044a36a2afaaf5531128d8
I20250626 19:57:39.580072 5438 ts_tablet_manager.cc:1939] T 94cefa59de044a36a2afaaf5531128d8 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:39.582302 5438 tablet_replica.cc:331] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:39.582850 5653 log.cc:1199] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/36997dab0ab3430bba63a90805759b30
I20250626 19:57:39.583482 5653 ts_tablet_manager.cc:1939] T 36997dab0ab3430bba63a90805759b30 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:39.588413 5664 consensus_queue.cc:1035] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:39.591960 5653 tablet_replica.cc:331] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:39.592815 5653 raft_consensus.cc:2241] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:39.593513 5653 pending_rounds.cc:70] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6: Trying to abort 1 pending ops.
I20250626 19:57:39.593768 5653 pending_rounds.cc:77] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964350465155072 op_type: NO_OP noop_request { }
I20250626 19:57:39.594122 5653 raft_consensus.cc:2887] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:57:39.594483 5653 raft_consensus.cc:2270] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:39.573714 5410 tablet_bootstrap.cc:654] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:39.597884 5664 consensus_queue.cc:1035] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:39.611698 5653 ts_tablet_manager.cc:1905] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:39.612300 5353 tablet_bootstrap.cc:654] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:39.616277 5433 ts_tablet_manager.cc:1918] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:39.616640 5433 log.cc:1199] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/7391b9eaea3e426ba4118fc8b53b72a6
I20250626 19:57:39.617295 5433 ts_tablet_manager.cc:1939] T 7391b9eaea3e426ba4118fc8b53b72a6 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:39.634968 4649 tablet_service.cc:1515] Processing DeleteTablet for tablet a31f1a7739544706bff2a500d6026086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:34 UTC) from {username='slave'} at 127.0.0.1:49550
W20250626 19:57:39.565596 4385 catalog_manager.cc:5001] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): alter failed for tablet 3c03691d021e4ee19d5bb287ab633a3c (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]),no further retry: Not found: Tablet not found: 3c03691d021e4ee19d5bb287ab633a3c
W20250626 19:57:39.572268 4393 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 62) took 3405 ms (client timeout 1000 ms). Trace:
W20250626 19:57:39.644006 4385 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.AlterSchema -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 78469.4us
W20250626 19:57:39.643947 4393 rpcz_store.cc:269] 0626 19:57:36.166888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.208582 (+ 41694us) service_pool.cc:225] Handling call
0626 19:57:39.572198 (+3363616us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49827,"reported_tablets":50}
I20250626 19:57:39.645222 4393 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_126ea31ac2c2483e84cedd411a0247b4" } new_table_name: "default.Table_4708ce225be445acbb30bbcc243949bc" modify_external_catalogs: true
I20250626 19:57:39.645956 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 36997dab0ab3430bba63a90805759b30 (table default.Table_c010e84626544214a45a911f02c4d903 [id=1b4629c56e5c4bf3b69fabbfb10b8008]) successfully deleted
I20250626 19:57:39.648025 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 94cefa59de044a36a2afaaf5531128d8 (table default.Table_c580e8a6ee5d417caa2f7d6811a68349 [id=b8bf68764b2c4a94b7299fa273983e06]) successfully deleted
W20250626 19:57:39.584050 4392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 78) took 1260 ms (client timeout 1000 ms). Trace:
W20250626 19:57:39.649283 4392 rpcz_store.cc:269] 0626 19:57:38.324150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:38.387472 (+ 63322us) service_pool.cc:225] Handling call
0626 19:57:39.583978 (+1196506us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54333,"reported_tablets":49}
I20250626 19:57:39.650279 4392 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_a5a4503fbcda4a7f95c722aaa5b4b0ce" } new_table_name: "default.Table_8b997467fa764e8fbf273c3b1f2ccb3e" modify_external_catalogs: true
I20250626 19:57:39.655692 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet fd6543699e254400b3516aece4486fe0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1474bfa74a6346bf9e3ff49019f6c9b3) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.634445 4390 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_38eec4fd44e848378efabe2ebbc88983" } modify_external_catalogs: true
I20250626 19:57:39.656800 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet 82823fd9ceb1485386616319fa7aeb43 with delete_type TABLET_DATA_DELETED (replaced by tablet f6ec952321924098a142f1438a283aab) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:39.656837 4942 tablet_service.cc:1468] Processing CreateTablet for tablet fafbf98e7f4544b6ad96c1c5ab6c82bc (DEFAULT_TABLE table=default.Table_c010e84626544214a45a911f02c4d903 [id=1b4629c56e5c4bf3b69fabbfb10b8008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.657261 4390 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_38eec4fd44e848378efabe2ebbc88983" } modify_external_catalogs: true
I20250626 19:57:39.657902 4942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fafbf98e7f4544b6ad96c1c5ab6c82bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.659268 5653 ts_tablet_manager.cc:1918] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:39.659624 5653 log.cc:1199] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/82823fd9ceb1485386616319fa7aeb43
I20250626 19:57:39.659476 4951 tablet_service.cc:1468] Processing CreateTablet for tablet e4aee96ad3454d65b974c867605c4cdc (DEFAULT_TABLE table=default.Table_6b965a2deba74eeeb9601ca4f736d338 [id=c4fa7a1654c649d38a23290cd19eaade]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.660104 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet fd6543699e254400b3516aece4486fe0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1474bfa74a6346bf9e3ff49019f6c9b3) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:39.660332 5653 ts_tablet_manager.cc:1939] T 82823fd9ceb1485386616319fa7aeb43 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:39.660483 4951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4aee96ad3454d65b974c867605c4cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.661343 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet c92deaa237064dee8f866e3a7a355ca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:34 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:39.664892 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet 36997dab0ab3430bba63a90805759b30 with delete_type TABLET_DATA_DELETED (replaced by tablet fafbf98e7f4544b6ad96c1c5ab6c82bc) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:39.666978 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:39.668716 4652 tablet_service.cc:1468] Processing CreateTablet for tablet fafbf98e7f4544b6ad96c1c5ab6c82bc (DEFAULT_TABLE table=default.Table_c010e84626544214a45a911f02c4d903 [id=1b4629c56e5c4bf3b69fabbfb10b8008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.669744 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fafbf98e7f4544b6ad96c1c5ab6c82bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.670694 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 82823fd9ceb1485386616319fa7aeb43 (table default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]) successfully deleted
W20250626 19:57:39.671375 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 2ab0320af47049cca3ca87172b9fe5e5 because the tablet was not found. No further retry: Not found: Tablet not found: 2ab0320af47049cca3ca87172b9fe5e5
W20250626 19:57:39.674211 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet cb34ffcd4da04e32a1220b3f298ec22d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb34ffcd4da04e32a1220b3f298ec22d already in progress: creating tablet
W20250626 19:57:39.674790 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 94cefa59de044a36a2afaaf5531128d8 because the tablet was not found. No further retry: Not found: Tablet not found: 94cefa59de044a36a2afaaf5531128d8
W20250626 19:57:39.675304 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 94327cab01df4338aaec157736d106f6 because the tablet was not found. No further retry: Not found: Tablet not found: 94327cab01df4338aaec157736d106f6
W20250626 19:57:39.676139 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet af58f34e2248432fae5c084bd3388028 because the tablet was not found. No further retry: Not found: Tablet not found: af58f34e2248432fae5c084bd3388028
W20250626 19:57:39.677006 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 78a8f494f8614e7ebb8ca880b6c80e78 because the tablet was not found. No further retry: Not found: Tablet not found: 78a8f494f8614e7ebb8ca880b6c80e78
W20250626 19:57:39.677582 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 499f6ae3262940b4859c14a53722482f because the tablet was not found. No further retry: Not found: Tablet not found: 499f6ae3262940b4859c14a53722482f
W20250626 19:57:39.678176 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 1cb6bda529624ac5b5a87a03b4a02d56 because the tablet was not found. No further retry: Not found: Tablet not found: 1cb6bda529624ac5b5a87a03b4a02d56
I20250626 19:57:39.678570 4391 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet dd0ca2fab40f45e5a1f9bd6059ca1f08
W20250626 19:57:39.678787 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 4b3bed19e7e34d24b995840a3bde5ef0 because the tablet was not found. No further retry: Not found: Tablet not found: 4b3bed19e7e34d24b995840a3bde5ef0
I20250626 19:57:39.679821 4391 catalog_manager.cc:6652] ReplaceTablet: tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 deleted and replaced by tablet 4a6058961bc7450ab489b26158e7dd17
W20250626 19:57:39.679780 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet a31f1a7739544706bff2a500d6026086 because the tablet was not found. No further retry: Not found: Tablet not found: a31f1a7739544706bff2a500d6026086
I20250626 19:57:39.680177 4391 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34796 (request call id 125) took 1157 ms. Trace:
W20250626 19:57:39.680531 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet fd6543699e254400b3516aece4486fe0 because the tablet was not found. No further retry: Not found: Tablet not found: fd6543699e254400b3516aece4486fe0
I20250626 19:57:39.681885 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a6058961bc7450ab489b26158e7dd17) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.682951 5653 tablet_replica.cc:331] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:39.683816 5653 raft_consensus.cc:2241] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:39.684360 5653 raft_consensus.cc:2270] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:39.680442 4391 rpcz_store.cc:276] 0626 19:57:38.522332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.488137 (+965805us) service_pool.cc:225] Handling call
0626 19:57:39.680126 (+191989us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:39.625446 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:39.625849 (+ 403us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:39.625876 (+ 27us) write_op.cc:435] Acquired schema lock
0626 19:57:39.625900 (+ 24us) tablet.cc:660] Decoding operations
0626 19:57:39.626203 (+ 303us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:39.626243 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:39.626269 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:39.626469 (+ 200us) tablet.cc:699] Row locks acquired
0626 19:57:39.626485 (+ 16us) write_op.cc:260] PREPARE: finished
0626 19:57:39.626590 (+ 105us) write_op.cc:270] Start()
0626 19:57:39.626673 (+ 83us) write_op.cc:276] Timestamp: P: 1750967859626601 usec, L: 0
0626 19:57:39.626693 (+ 20us) op_driver.cc:348] REPLICATION: starting
0626 19:57:39.627233 (+ 540us) log.cc:844] Serialized 659 byte log entry
0626 19:57:39.672287 (+ 45054us) op_driver.cc:464] REPLICATION: finished
0626 19:57:39.674432 (+ 2145us) write_op.cc:301] APPLY: starting
0626 19:57:39.674478 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:39.674573 (+ 95us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:39.674658 (+ 85us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:39.677190 (+ 2532us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:39.677403 (+ 213us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:39.677459 (+ 56us) write_op.cc:312] APPLY: finished
0626 19:57:39.677654 (+ 195us) log.cc:844] Serialized 29 byte log entry
0626 19:57:39.677795 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:39.677858 (+ 63us) write_op.cc:454] Released schema lock
0626 19:57:39.677888 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2028,"mutex_wait_us":81951,"num_ops":2,"prepare.queue_time_us":83145,"prepare.run_cpu_time_us":2049,"prepare.run_wall_time_us":4091,"raft.queue_time_us":996,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":13,"replication_time_us":45522,"threads_started":1}]]}
I20250626 19:57:39.686580 5653 ts_tablet_manager.cc:1905] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:39.535297 4399 rpcz_store.cc:269] 0626 19:57:36.463240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:36.467927 (+ 4687us) service_pool.cc:225] Handling call
0626 19:57:39.479545 (+3011618us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:37.371075 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:37.371443 (+ 368us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:37.371469 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:37.371483 (+ 14us) tablet.cc:660] Decoding operations
0626 19:57:37.371741 (+ 258us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:37.371779 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:37.371802 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:37.371931 (+ 129us) tablet.cc:699] Row locks acquired
0626 19:57:37.371943 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:37.372048 (+ 105us) write_op.cc:270] Start()
0626 19:57:37.372125 (+ 77us) write_op.cc:276] Timestamp: P: 1750967857372174 usec, L: 0
0626 19:57:37.372139 (+ 14us) op_driver.cc:348] REPLICATION: starting
0626 19:57:37.372602 (+ 463us) log.cc:844] Serialized 530 byte log entry
0626 19:57:37.407266 (+ 34664us) op_driver.cc:464] REPLICATION: finished
0626 19:57:37.407662 (+ 396us) write_op.cc:301] APPLY: starting
0626 19:57:37.407736 (+ 74us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:37.407828 (+ 92us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:37.407924 (+ 96us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:37.408078 (+ 154us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:37.408218 (+ 140us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:37.408264 (+ 46us) write_op.cc:312] APPLY: finished
0626 19:57:37.408434 (+ 170us) log.cc:844] Serialized 23 byte log entry
0626 19:57:37.408591 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:37.408656 (+ 65us) write_op.cc:454] Released schema lock
0626 19:57:37.408698 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:37.408907 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:37.408939 (+ 32us) write_op.cc:454] Released schema lock
Related trace 'op':
0626 19:57:37.491236 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:37.491663 (+ 427us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:37.491688 (+ 25us) write_op.cc:435] Acquired schema lock
0626 19:57:37.491700 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:37.491945 (+ 245us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:37.491987 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:37.492012 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:37.492155 (+ 143us) tablet.cc:699] Row locks acquired
0626 19:57:37.492181 (+ 26us) write_op.cc:260] PREPARE: finished
0626 19:57:37.492295 (+ 114us) write_op.cc:270] Start()
0626 19:57:37.492415 (+ 120us) write_op.cc:276] Timestamp: P: 1750967857492270 usec, L: 0
0626 19:57:37.492432 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:37.493227 (+ 795us) log.cc:844] Serialized 356 byte log entry
0626 19:57:37.542435 (+ 49208us) op_driver.cc:464] REPLICATION: finished
0626 19:57:37.543710 (+ 1275us) write_op.cc:301] APPLY: starting
0626 19:57:37.543766 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:37.544166 (+ 400us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:37.544265 (+ 99us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:37.544458 (+ 193us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:37.544596 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:37.544644 (+ 48us) write_op.cc:312] APPLY: finished
0626 19:57:37.544815 (+ 171us) log.cc:844] Serialized 23 byte log entry
0626 19:57:37.544970 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:37.545033 (+ 63us) write_op.cc:454] Released schema lock
0626 19:57:37.545066 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":229639,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":237,"apply.run_cpu_time_us":1223,"apply.run_wall_time_us":1222,"num_ops":1,"prepare.queue_time_us":7524,"prepare.run_cpu_time_us":1804,"prepare.run_wall_time_us":1810,"raft.queue_time_us":1087,"raft.run_cpu_time_us":616,"raft.run_wall_time_us":617,"replication_time_us":35024}],["op",{"apply.queue_time_us":125,"apply.run_cpu_time_us":1276,"apply.run_wall_time_us":1974,"num_ops":1,"prepare.queue_time_us":13183,"prepare.run_cpu_time_us":2303,"prepare.run_wall_time_us":3359,"raft.queue_time_us":82,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":11,"replication_time_us":49835}]]}
I20250626 19:57:39.692561 4398 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet f43aed8f07844c5ca727ae6df63896df
I20250626 19:57:39.695519 4398 catalog_manager.cc:6652] ReplaceTablet: tablet f43aed8f07844c5ca727ae6df63896df deleted and replaced by tablet 075dbe46bab14d40a8008ce5c24bcf38
I20250626 19:57:39.600217 4800 tablet_service.cc:1468] Processing CreateTablet for tablet e94a73b50b564a97bbee0f5015bb8805 (DEFAULT_TABLE table=default.Table_fbcc45f0e38f4ed184900c0f58067165 [id=10f3db9788ba4edda2aab300e7313abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.571630 5700 tablet.cc:1722] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:39.663771 5429 consensus_queue.cc:1035] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:39.684720 4794 tablet_service.cc:1468] Processing CreateTablet for tablet fafbf98e7f4544b6ad96c1c5ab6c82bc (DEFAULT_TABLE table=default.Table_c010e84626544214a45a911f02c4d903 [id=1b4629c56e5c4bf3b69fabbfb10b8008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.701232 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet f43aed8f07844c5ca727ae6df63896df with delete_type TABLET_DATA_DELETED (replaced by tablet 075dbe46bab14d40a8008ce5c24bcf38) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.703284 4794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fafbf98e7f4544b6ad96c1c5ab6c82bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.711503 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e94a73b50b564a97bbee0f5015bb8805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.717445 5438 raft_consensus.cc:2241] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:39.723070 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet c92deaa237064dee8f866e3a7a355ca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:34 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:39.728197 4398 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34796 (request call id 126) took 1085 ms. Trace:
I20250626 19:57:39.729768 5366 tablet_bootstrap.cc:492] T 6cc7a336385f499ca19566d5f2901e98 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:39.736111 5366 ts_tablet_manager.cc:1397] T 6cc7a336385f499ca19566d5f2901e98 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.300s user 0.038s sys 0.028s
I20250626 19:57:39.738348 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:39.738314 5366 raft_consensus.cc:357] T 6cc7a336385f499ca19566d5f2901e98 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.739001 5366 raft_consensus.cc:383] T 6cc7a336385f499ca19566d5f2901e98 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:39.739241 5366 raft_consensus.cc:738] T 6cc7a336385f499ca19566d5f2901e98 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:39.741633 5366 consensus_queue.cc:260] T 6cc7a336385f499ca19566d5f2901e98 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.742110 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:39.743865 5366 ts_tablet_manager.cc:1428] T 6cc7a336385f499ca19566d5f2901e98 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250626 19:57:39.744658 5366 tablet_bootstrap.cc:492] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:39.748765 5438 raft_consensus.cc:2270] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:39.757030 5653 ts_tablet_manager.cc:1918] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:39.757402 5653 log.cc:1199] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/dd0ca2fab40f45e5a1f9bd6059ca1f08
I20250626 19:57:39.758076 5653 ts_tablet_manager.cc:1939] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:39.755911 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a6058961bc7450ab489b26158e7dd17) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.758878 5366 tablet_bootstrap.cc:654] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:39.765033 4797 tablet_service.cc:1468] Processing CreateTablet for tablet d8399c219d0e4485bb7926eefd78f210 (DEFAULT_TABLE table=default.Table_e5f9396b41eb4c5eb98ec42ec48d7cdc [id=bad1d85ef3a347ab9abe78289306ca4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.765358 4611 leader_election.cc:304] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:39.766084 4797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8399c219d0e4485bb7926eefd78f210. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.766371 5689 raft_consensus.cc:2747] T e60b4d53c40c4dffab928f0e3fb4a80b P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:57:39.767494 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet f43aed8f07844c5ca727ae6df63896df with delete_type TABLET_DATA_DELETED (replaced by tablet 075dbe46bab14d40a8008ce5c24bcf38) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:39.773870 4395 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 124) took 1617 ms. Trace:
I20250626 19:57:39.774197 4395 rpcz_store.cc:276] 0626 19:57:38.156671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.462457 (+1305786us) service_pool.cc:225] Handling call
0626 19:57:39.522953 (+ 60496us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:39.561745 (+ 38792us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:39.773281 (+211536us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:39.773567 (+ 286us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:39.773813 (+ 246us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:39.629635 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:39.630048 (+ 413us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:39.630073 (+ 25us) write_op.cc:435] Acquired schema lock
0626 19:57:39.630091 (+ 18us) tablet.cc:660] Decoding operations
0626 19:57:39.630366 (+ 275us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:39.630405 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:39.630432 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:39.630624 (+ 192us) tablet.cc:699] Row locks acquired
0626 19:57:39.630640 (+ 16us) write_op.cc:260] PREPARE: finished
0626 19:57:39.630897 (+ 257us) write_op.cc:270] Start()
0626 19:57:39.630976 (+ 79us) write_op.cc:276] Timestamp: P: 1750967859630902 usec, L: 0
0626 19:57:39.630993 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:39.631430 (+ 437us) log.cc:844] Serialized 433 byte log entry
0626 19:57:39.768907 (+137477us) op_driver.cc:464] REPLICATION: finished
0626 19:57:39.769344 (+ 437us) write_op.cc:301] APPLY: starting
0626 19:57:39.769403 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:39.769505 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:39.769632 (+ 127us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:39.771998 (+ 2366us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:39.772133 (+ 135us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:39.772175 (+ 42us) write_op.cc:312] APPLY: finished
0626 19:57:39.772352 (+ 177us) log.cc:844] Serialized 29 byte log entry
0626 19:57:39.773001 (+ 649us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:39.773068 (+ 67us) write_op.cc:454] Released schema lock
0626 19:57:39.773100 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":292,"mutex_wait_us":52354,"num_ops":2,"prepare.queue_time_us":56664,"prepare.run_cpu_time_us":1870,"prepare.run_wall_time_us":2802,"replication_time_us":137818}]]}
I20250626 19:57:39.776454 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:39.783368 4399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_987278d268e04477828ad87c6d9c8c01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:39.784927 4399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_987278d268e04477828ad87c6d9c8c01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:57:39.788667 4609 consensus_peers.cc:487] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 -> Peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Couldn't send request to peer 93f885c6802341c889a40e9c20fbfff1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94cefa59de044a36a2afaaf5531128d8. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:39.791193 4395 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_dcae42312225483a9c5104bf2551f652"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:39.793520 4395 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcae42312225483a9c5104bf2551f652 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:39.795385 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_56d52ecce53b4f1e8afea2f6d1695a92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:39.796826 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56d52ecce53b4f1e8afea2f6d1695a92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:39.816268 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 (table default.Table_889f18064aab4774b17bfa478a0a7e33 [id=0df063bc889246c3ba841b1b2f8e6104]) successfully deleted
W20250626 19:57:39.818583 4912 consensus_peers.cc:487] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:39.819407 4611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 53044.8us
W20250626 19:57:39.821288 4611 consensus_peers.cc:487] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94cefa59de044a36a2afaaf5531128d8. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:39.822669 4649 tablet_service.cc:1515] Processing DeleteTablet for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a6058961bc7450ab489b26158e7dd17) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:39.823621 4649 tablet_service.cc:1515] Processing DeleteTablet for tablet f43aed8f07844c5ca727ae6df63896df with delete_type TABLET_DATA_DELETED (replaced by tablet 075dbe46bab14d40a8008ce5c24bcf38) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:39.825735 5410 tablet_bootstrap.cc:492] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:39.826141 5410 ts_tablet_manager.cc:1397] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.296s user 0.006s sys 0.004s
I20250626 19:57:39.826184 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 86) took 1878 ms. Trace:
I20250626 19:57:39.831240 5653 tablet_replica.cc:331] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:39.832067 5653 raft_consensus.cc:2241] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:39.835248 5653 raft_consensus.cc:2270] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:39.836081 5353 tablet_bootstrap.cc:492] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:39.836704 5353 ts_tablet_manager.cc:1397] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.476s user 0.008s sys 0.003s
I20250626 19:57:39.839466 5653 ts_tablet_manager.cc:1905] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:39.842046 5410 raft_consensus.cc:357] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.763804 4799 tablet_service.cc:1468] Processing CreateTablet for tablet e4aee96ad3454d65b974c867605c4cdc (DEFAULT_TABLE table=default.Table_6b965a2deba74eeeb9601ca4f736d338 [id=c4fa7a1654c649d38a23290cd19eaade]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.851974 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4aee96ad3454d65b974c867605c4cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.856040 5410 raft_consensus.cc:383] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:39.856292 5410 raft_consensus.cc:738] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:39.826632 5433 rpcz_store.cc:276] 0626 19:57:37.948231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:37.948419 (+ 188us) service_pool.cc:225] Handling call
0626 19:57:39.082969 (+1134550us) ts_tablet_manager.cc:1145] Deleting tablet 7391b9eaea3e426ba4118fc8b53b72a6
0626 19:57:39.083061 (+ 92us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:39.484330 (+401269us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:39.616049 (+131719us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:39.826128 (+210079us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1920,"tablet-delete.queue_time_us":1131664}
I20250626 19:57:39.857199 5433 tablet_replica.cc:331] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:39.857859 5433 raft_consensus.cc:2241] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:39.858264 5433 raft_consensus.cc:2270] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:39.860352 5433 ts_tablet_manager.cc:1905] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:39.867831 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 7391b9eaea3e426ba4118fc8b53b72a6 (table default.Table_e5f9396b41eb4c5eb98ec42ec48d7cdc [id=bad1d85ef3a347ab9abe78289306ca4d]) successfully deleted
I20250626 19:57:39.876995 4957 tablet_service.cc:1468] Processing CreateTablet for tablet f807cf54bbb146ff989762c18410e6bb (DEFAULT_TABLE table=default.Table_f85ad493ebbf4e3b8c7eea1edfcf6a67 [id=fa2e7940491b4fbbac3ee7d93df1476d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.762843 5438 ts_tablet_manager.cc:1905] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:39.888768 5653 ts_tablet_manager.cc:1918] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:39.889189 5653 log.cc:1199] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/f43aed8f07844c5ca727ae6df63896df
I20250626 19:57:39.889930 5653 ts_tablet_manager.cc:1939] T f43aed8f07844c5ca727ae6df63896df P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:39.891585 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f807cf54bbb146ff989762c18410e6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.891342 5353 raft_consensus.cc:357] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.891973 5353 raft_consensus.cc:383] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:39.892215 5353 raft_consensus.cc:738] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:39.892843 5353 consensus_queue.cc:260] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.900907 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet f43aed8f07844c5ca727ae6df63896df (table default.Table_d6eed56e9d514833bfe29ff3aba9718a [id=07c980b0511a43c0aa406597788cd05b]) successfully deleted
W20250626 19:57:39.902441 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet c92deaa237064dee8f866e3a7a355ca4 because the tablet was not found. No further retry: Not found: Tablet not found: c92deaa237064dee8f866e3a7a355ca4
I20250626 19:57:39.903820 4795 tablet_service.cc:1468] Processing CreateTablet for tablet f807cf54bbb146ff989762c18410e6bb (DEFAULT_TABLE table=default.Table_f85ad493ebbf4e3b8c7eea1edfcf6a67 [id=fa2e7940491b4fbbac3ee7d93df1476d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.855540 4651 tablet_service.cc:1468] Processing CreateTablet for tablet e4aee96ad3454d65b974c867605c4cdc (DEFAULT_TABLE table=default.Table_6b965a2deba74eeeb9601ca4f736d338 [id=c4fa7a1654c649d38a23290cd19eaade]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.913167 4651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4aee96ad3454d65b974c867605c4cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.913727 4952 tablet_service.cc:1468] Processing CreateTablet for tablet d8399c219d0e4485bb7926eefd78f210 (DEFAULT_TABLE table=default.Table_e5f9396b41eb4c5eb98ec42ec48d7cdc [id=bad1d85ef3a347ab9abe78289306ca4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.853590 4648 tablet_service.cc:1468] Processing CreateTablet for tablet d8399c219d0e4485bb7926eefd78f210 (DEFAULT_TABLE table=default.Table_e5f9396b41eb4c5eb98ec42ec48d7cdc [id=bad1d85ef3a347ab9abe78289306ca4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.914870 4952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8399c219d0e4485bb7926eefd78f210. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.916983 4648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8399c219d0e4485bb7926eefd78f210. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.862409 5410 consensus_queue.cc:260] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.902956 4656 tablet_service.cc:1468] Processing CreateTablet for tablet f807cf54bbb146ff989762c18410e6bb (DEFAULT_TABLE table=default.Table_f85ad493ebbf4e3b8c7eea1edfcf6a67 [id=fa2e7940491b4fbbac3ee7d93df1476d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.736114 4398 rpcz_store.cc:276] 0626 19:57:38.642874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.531891 (+889017us) service_pool.cc:225] Handling call
0626 19:57:39.728115 (+196224us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:39.622108 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:39.622469 (+ 361us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:39.622491 (+ 22us) write_op.cc:435] Acquired schema lock
0626 19:57:39.622503 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:39.622835 (+ 332us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:39.622875 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:39.622904 (+ 29us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:39.623086 (+ 182us) tablet.cc:699] Row locks acquired
0626 19:57:39.623098 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:39.623195 (+ 97us) write_op.cc:270] Start()
0626 19:57:39.623276 (+ 81us) write_op.cc:276] Timestamp: P: 1750967859623208 usec, L: 0
0626 19:57:39.623291 (+ 15us) op_driver.cc:348] REPLICATION: starting
0626 19:57:39.623835 (+ 544us) log.cc:844] Serialized 659 byte log entry
0626 19:57:39.672063 (+ 48228us) op_driver.cc:464] REPLICATION: finished
0626 19:57:39.672736 (+ 673us) write_op.cc:301] APPLY: starting
0626 19:57:39.672791 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:39.672924 (+ 133us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:39.673017 (+ 93us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:39.673317 (+ 300us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:39.673506 (+ 189us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:39.673547 (+ 41us) write_op.cc:312] APPLY: finished
0626 19:57:39.673735 (+ 188us) log.cc:844] Serialized 29 byte log entry
0626 19:57:39.673879 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:39.673942 (+ 63us) write_op.cc:454] Released schema lock
0626 19:57:39.673973 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:39.674174 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:39.674201 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":516,"apply.run_cpu_time_us":1418,"apply.run_wall_time_us":1417,"num_ops":2,"prepare.queue_time_us":86898,"prepare.run_cpu_time_us":2023,"prepare.run_wall_time_us":2021,"raft.queue_time_us":4417,"raft.run_cpu_time_us":793,"raft.run_wall_time_us":796,"replication_time_us":48681}]]}
I20250626 19:57:39.923300 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f807cf54bbb146ff989762c18410e6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.927523 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f807cf54bbb146ff989762c18410e6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.940740 5688 raft_consensus.cc:491] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:39.941296 5688 raft_consensus.cc:513] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:39.943378 5688 leader_election.cc:290] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:39.944124 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5706841a14bd4ece965098b88ccb7ec8" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:39.947669 4756 leader_election.cc:343] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:39.952657 4383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250626 19:57:39.956061 4383 consensus_peers.cc:487] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af -> Peer e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823): Couldn't send request to peer e6c58006b6be41c1bd6bdb5eb010d360. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.190:33823: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250626 19:57:39.900669 5353 ts_tablet_manager.cc:1428] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.064s user 0.002s sys 0.003s
I20250626 19:57:39.964509 5438 ts_tablet_manager.cc:1918] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:39.968341 5353 tablet_bootstrap.cc:492] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:39.971877 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5706841a14bd4ece965098b88ccb7ec8" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:39.973171 4912 consensus_peers.cc:487] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:39.974355 4755 leader_election.cc:343] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:39.977706 5438 log.cc:1199] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/94327cab01df4338aaec157736d106f6
I20250626 19:57:39.977733 4755 leader_election.cc:304] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:39.978480 5438 ts_tablet_manager.cc:1939] T 94327cab01df4338aaec157736d106f6 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:39.979113 5688 raft_consensus.cc:2747] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:39.979341 5410 ts_tablet_manager.cc:1428] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.153s user 0.007s sys 0.000s
I20250626 19:57:39.980626 5410 tablet_bootstrap.cc:492] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:39.981338 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_8ef072e9d240402397d79db39729e650"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:39.982823 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ef072e9d240402397d79db39729e650 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:39.984027 4675 raft_consensus.cc:3058] T 419c4018256546ef9431ac2db619ee33 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:39.986891 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 94327cab01df4338aaec157736d106f6 (table default.Table_beec4e8c60ae4e6880c0ff709d3c0ff3 [id=6e84598eef744e8c9b1d918036bc67ec]) successfully deleted
I20250626 19:57:39.987958 4942 tablet_service.cc:1468] Processing CreateTablet for tablet fc526a05990446a3b3bc973e79511476 (DEFAULT_TABLE table=default.Table_38eec4fd44e848378efabe2ebbc88983 [id=4e600070f9e44758a4b11f0adeab28e5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250626 19:57:39.988390 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 1cb6bda529624ac5b5a87a03b4a02d56 because the tablet was not found. No further retry: Not found: Tablet not found: 1cb6bda529624ac5b5a87a03b4a02d56
I20250626 19:57:39.989076 4942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc526a05990446a3b3bc973e79511476. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:39.990058 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet a31f1a7739544706bff2a500d6026086 because the tablet was not found. No further retry: Not found: Tablet not found: a31f1a7739544706bff2a500d6026086
I20250626 19:57:39.990056 4801 tablet_service.cc:1468] Processing CreateTablet for tablet fc526a05990446a3b3bc973e79511476 (DEFAULT_TABLE table=default.Table_38eec4fd44e848378efabe2ebbc88983 [id=4e600070f9e44758a4b11f0adeab28e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.990854 5438 tablet_replica.cc:331] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:39.991058 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc526a05990446a3b3bc973e79511476. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:39.991568 5438 raft_consensus.cc:2241] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:39.991981 5438 raft_consensus.cc:2270] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:39.993527 5438 ts_tablet_manager.cc:1905] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:39.996505 4951 tablet_service.cc:1468] Processing CreateTablet for tablet 6d43617ed2cb4a0085e59eefcf8b19f1 (DEFAULT_TABLE table=default.Table_3cb641007af847a68d8f13f80829fe0e [id=fa6062d58cf542f78b26136fda915a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:39.997603 4951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d43617ed2cb4a0085e59eefcf8b19f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:40.004325 5702 raft_consensus.cc:491] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:40.004803 5702 raft_consensus.cc:513] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.007481 5702 leader_election.cc:290] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:40.008282 4649 tablet_service.cc:1468] Processing CreateTablet for tablet fc526a05990446a3b3bc973e79511476 (DEFAULT_TABLE table=default.Table_38eec4fd44e848378efabe2ebbc88983 [id=4e600070f9e44758a4b11f0adeab28e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:40.008455 4653 tablet_service.cc:1468] Processing CreateTablet for tablet 6d43617ed2cb4a0085e59eefcf8b19f1 (DEFAULT_TABLE table=default.Table_3cb641007af847a68d8f13f80829fe0e [id=fa6062d58cf542f78b26136fda915a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:40.008948 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6cc7a336385f499ca19566d5f2901e98" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:40.009390 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc526a05990446a3b3bc973e79511476. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:40.009503 4815 raft_consensus.cc:2466] T 6cc7a336385f499ca19566d5f2901e98 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:40.011292 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d43617ed2cb4a0085e59eefcf8b19f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:40.011109 4911 leader_election.cc:304] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:40.015424 5366 tablet_bootstrap.cc:492] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:40.015842 5366 ts_tablet_manager.cc:1397] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.271s user 0.034s sys 0.038s
I20250626 19:57:40.018270 5366 raft_consensus.cc:357] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.019088 5366 raft_consensus.cc:383] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.019335 5366 raft_consensus.cc:738] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:40.019918 5366 consensus_queue.cc:260] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.021875 5366 ts_tablet_manager.cc:1428] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250626 19:57:40.033453 5438 ts_tablet_manager.cc:1918] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:40.033852 5438 log.cc:1199] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/82823fd9ceb1485386616319fa7aeb43
I20250626 19:57:40.034475 5438 ts_tablet_manager.cc:1939] T 82823fd9ceb1485386616319fa7aeb43 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:40.037269 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 82823fd9ceb1485386616319fa7aeb43 (table default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]) successfully deleted
I20250626 19:57:40.037554 5438 tablet_replica.cc:331] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
W20250626 19:57:40.037954 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet c92deaa237064dee8f866e3a7a355ca4 because the tablet was not found. No further retry: Not found: Tablet not found: c92deaa237064dee8f866e3a7a355ca4
I20250626 19:57:40.038281 5438 raft_consensus.cc:2241] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250626 19:57:40.038467 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 3c03691d021e4ee19d5bb287ab633a3c because the tablet was not found. No further retry: Not found: Tablet not found: 3c03691d021e4ee19d5bb287ab633a3c
I20250626 19:57:40.038698 5438 raft_consensus.cc:2270] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:40.040323 5438 ts_tablet_manager.cc:1905] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:40.044358 4395 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_ab8780ef94764aadbd99279e11543fd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:40.045876 4395 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab8780ef94764aadbd99279e11543fd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:40.047803 4390 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet fc526a05990446a3b3bc973e79511476
I20250626 19:57:40.049597 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet fc526a05990446a3b3bc973e79511476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:40.050575 4793 tablet_service.cc:1515] Processing DeleteTablet for tablet fc526a05990446a3b3bc973e79511476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:40.056186 4793 tablet_service.cc:1468] Processing CreateTablet for tablet 6d43617ed2cb4a0085e59eefcf8b19f1 (DEFAULT_TABLE table=default.Table_3cb641007af847a68d8f13f80829fe0e [id=fa6062d58cf542f78b26136fda915a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:40.057248 4793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d43617ed2cb4a0085e59eefcf8b19f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:40.073813 5438 ts_tablet_manager.cc:1918] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:40.074184 5438 log.cc:1199] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/36997dab0ab3430bba63a90805759b30
I20250626 19:57:40.074815 5438 ts_tablet_manager.cc:1939] T 36997dab0ab3430bba63a90805759b30 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:40.079874 5366 tablet_bootstrap.cc:492] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:40.080946 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 36997dab0ab3430bba63a90805759b30 (table default.Table_c010e84626544214a45a911f02c4d903 [id=1b4629c56e5c4bf3b69fabbfb10b8008]) successfully deleted
W20250626 19:57:40.083214 4383 catalog_manager.cc:4908] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet cb34ffcd4da04e32a1220b3f298ec22d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb34ffcd4da04e32a1220b3f298ec22d already in progress: creating tablet
W20250626 19:57:40.084508 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 2ab0320af47049cca3ca87172b9fe5e5 because the tablet was not found. No further retry: Not found: Tablet not found: 2ab0320af47049cca3ca87172b9fe5e5
W20250626 19:57:40.085726 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 7391b9eaea3e426ba4118fc8b53b72a6 because the tablet was not found. No further retry: Not found: Tablet not found: 7391b9eaea3e426ba4118fc8b53b72a6
W20250626 19:57:40.086917 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 499f6ae3262940b4859c14a53722482f because the tablet was not found. No further retry: Not found: Tablet not found: 499f6ae3262940b4859c14a53722482f
W20250626 19:57:40.088199 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 5785508e37624f02bc2f6cc93007e3f8 because the tablet was not found. No further retry: Not found: Tablet not found: 5785508e37624f02bc2f6cc93007e3f8
I20250626 19:57:40.063443 5410 tablet_bootstrap.cc:654] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:40.091082 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet fd6543699e254400b3516aece4486fe0 because the tablet was not found. No further retry: Not found: Tablet not found: fd6543699e254400b3516aece4486fe0
W20250626 19:57:40.092271 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 4b3bed19e7e34d24b995840a3bde5ef0 because the tablet was not found. No further retry: Not found: Tablet not found: 4b3bed19e7e34d24b995840a3bde5ef0
I20250626 19:57:39.976688 4394 catalog_manager.cc:5582] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:40.012313 5664 raft_consensus.cc:2802] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250626 19:57:40.098227 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet af58f34e2248432fae5c084bd3388028 because the tablet was not found. No further retry: Not found: Tablet not found: af58f34e2248432fae5c084bd3388028
I20250626 19:57:40.100157 5438 tablet_replica.cc:331] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:40.102463 5438 raft_consensus.cc:2241] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:40.103221 5438 raft_consensus.cc:2270] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:40.097808 5664 raft_consensus.cc:491] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:40.103319 5366 tablet_bootstrap.cc:654] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.118086 5702 consensus_queue.cc:1035] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:40.119675 5438 ts_tablet_manager.cc:1905] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:40.122742 4950 tablet_service.cc:1515] Processing DeleteTablet for tablet fc526a05990446a3b3bc973e79511476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:40.126014 4394 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:40.126860 4394 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:40.127827 4394 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 6b66cda4f8ed410184b434d5083deac4 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]). Expected version 1 got 0
I20250626 19:57:40.128932 5366 tablet_bootstrap.cc:492] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:40.129348 5366 ts_tablet_manager.cc:1397] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.000s
I20250626 19:57:40.136339 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6cc7a336385f499ca19566d5f2901e98" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:40.137359 5366 raft_consensus.cc:357] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.137995 5366 raft_consensus.cc:383] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.138226 5366 raft_consensus.cc:738] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
W20250626 19:57:40.138242 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet fc526a05990446a3b3bc973e79511476 because the tablet was not found. No further retry: Not found: Tablet not found: fc526a05990446a3b3bc973e79511476
I20250626 19:57:40.138804 5366 consensus_queue.cc:260] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:40.140400 4755 consensus_peers.cc:487] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:40.110541 5664 raft_consensus.cc:3058] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:40.151916 5438 ts_tablet_manager.cc:1918] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:40.152277 5438 log.cc:1199] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/dd0ca2fab40f45e5a1f9bd6059ca1f08
I20250626 19:57:40.152925 5366 ts_tablet_manager.cc:1428] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20250626 19:57:40.154042 5438 ts_tablet_manager.cc:1939] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:40.156059 5366 tablet_bootstrap.cc:492] T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:40.161911 5366 tablet_bootstrap.cc:654] T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.165231 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 (table default.Table_889f18064aab4774b17bfa478a0a7e33 [id=0df063bc889246c3ba841b1b2f8e6104]) successfully deleted
I20250626 19:57:40.172644 5438 tablet_replica.cc:331] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:40.173437 5438 raft_consensus.cc:2241] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:40.175659 5438 raft_consensus.cc:2270] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:40.177397 5438 ts_tablet_manager.cc:1905] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:40.192176 5366 tablet_bootstrap.cc:492] T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:40.192607 5366 ts_tablet_manager.cc:1397] T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.002s
I20250626 19:57:40.194183 5433 ts_tablet_manager.cc:1918] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250626 19:57:40.159161 4912 leader_election.cc:343] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:40.208698 5664 raft_consensus.cc:513] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.214811 5438 ts_tablet_manager.cc:1918] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:40.215193 5438 log.cc:1199] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/f43aed8f07844c5ca727ae6df63896df
I20250626 19:57:40.215873 5438 ts_tablet_manager.cc:1939] T f43aed8f07844c5ca727ae6df63896df P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:40.218015 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet f43aed8f07844c5ca727ae6df63896df (table default.Table_d6eed56e9d514833bfe29ff3aba9718a [id=07c980b0511a43c0aa406597788cd05b]) successfully deleted
I20250626 19:57:40.217288 5366 raft_consensus.cc:357] T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.219529 5366 raft_consensus.cc:383] T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.219792 5366 raft_consensus.cc:738] T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
W20250626 19:57:40.219858 4383 catalog_manager.cc:4908] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet fc526a05990446a3b3bc973e79511476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc526a05990446a3b3bc973e79511476 already in progress: creating tablet
I20250626 19:57:40.227787 5366 consensus_queue.cc:260] T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.232474 5366 ts_tablet_manager.cc:1428] T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.040s user 0.006s sys 0.001s
I20250626 19:57:40.233355 5366 tablet_bootstrap.cc:492] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:40.202157 5410 tablet_bootstrap.cc:492] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:40.242939 5410 ts_tablet_manager.cc:1397] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.262s user 0.010s sys 0.000s
I20250626 19:57:40.245299 5410 raft_consensus.cc:357] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.246014 5410 raft_consensus.cc:383] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.246306 5410 raft_consensus.cc:738] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:40.218683 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6cc7a336385f499ca19566d5f2901e98" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:40.247992 5410 consensus_queue.cc:260] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.250080 5410 ts_tablet_manager.cc:1428] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250626 19:57:40.250466 5366 tablet_bootstrap.cc:654] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.251062 5410 tablet_bootstrap.cc:492] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:40.255239 5433 log.cc:1199] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/3c03691d021e4ee19d5bb287ab633a3c
I20250626 19:57:40.256031 5433 ts_tablet_manager.cc:1939] T 3c03691d021e4ee19d5bb287ab633a3c P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:40.256803 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 92) took 1863 ms. Trace:
I20250626 19:57:40.257128 5433 rpcz_store.cc:276] 0626 19:57:38.409161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:38.421813 (+ 12652us) service_pool.cc:225] Handling call
0626 19:57:39.857057 (+1435244us) ts_tablet_manager.cc:1145] Deleting tablet 3c03691d021e4ee19d5bb287ab633a3c
0626 19:57:39.857101 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:40.095490 (+238389us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:40.193964 (+ 98474us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:40.256755 (+ 62791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1426052}
I20250626 19:57:40.259534 5433 tablet_replica.cc:331] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:40.259363 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 3c03691d021e4ee19d5bb287ab633a3c (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]) successfully deleted
W20250626 19:57:40.258960 4912 leader_election.cc:343] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:40.262455 5664 leader_election.cc:290] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:40.266265 5506 raft_consensus.cc:491] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:40.266723 5506 raft_consensus.cc:513] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.268517 5506 leader_election.cc:290] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:40.274834 5429 consensus_queue.cc:1035] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:40.277016 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ba78c1b40ef4925b4108e1455bcfc44" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:40.277529 5366 tablet_bootstrap.cc:492] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:40.277972 5366 ts_tablet_manager.cc:1397] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.001s
W20250626 19:57:40.282871 4756 leader_election.cc:343] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:40.287434 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_d51d8fa1843f4e47a913158768b4689c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:40.289024 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d51d8fa1843f4e47a913158768b4689c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:40.292868 4395 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_7490b89a33c842b0853726a36f5cebe6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:40.294667 4395 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7490b89a33c842b0853726a36f5cebe6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:40.270860 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ba78c1b40ef4925b4108e1455bcfc44" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:40.295845 4755 leader_election.cc:343] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:40.296208 4755 leader_election.cc:304] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:40.297056 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6cc7a336385f499ca19566d5f2901e98" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:40.297272 5506 raft_consensus.cc:2747] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:40.297572 4820 raft_consensus.cc:3058] T 6cc7a336385f499ca19566d5f2901e98 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:40.298507 5506 raft_consensus.cc:491] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:40.298924 5506 raft_consensus.cc:513] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.302096 5506 leader_election.cc:290] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:40.303309 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0bcbb044d1549c88e5b83eeb5c709af" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:40.304440 4820 raft_consensus.cc:2466] T 6cc7a336385f499ca19566d5f2901e98 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
W20250626 19:57:40.304591 4755 leader_election.cc:343] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:40.306829 4675 raft_consensus.cc:3058] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:40.310148 5664 consensus_queue.cc:1035] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:40.311446 4911 leader_election.cc:304] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:40.312451 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0bcbb044d1549c88e5b83eeb5c709af" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:40.313711 5506 consensus_queue.cc:1035] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:40.314474 5664 raft_consensus.cc:2802] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:40.314903 5664 raft_consensus.cc:695] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
W20250626 19:57:40.315400 4756 leader_election.cc:343] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:40.315763 4756 leader_election.cc:304] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:40.315665 5664 consensus_queue.cc:237] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.318034 5506 raft_consensus.cc:2747] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:40.339324 5433 raft_consensus.cc:2241] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:40.340123 5433 raft_consensus.cc:2270] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:40.341909 5433 ts_tablet_manager.cc:1905] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:40.345662 4394 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:40.342166 5366 raft_consensus.cc:357] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.347555 5366 raft_consensus.cc:383] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.347826 5366 raft_consensus.cc:738] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:40.350672 4395 master_service.cc:948] ReplaceTablet: received request to replace tablet 6b8473259485460783f62f8193d5b02c from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:40.362680 5506 consensus_queue.cc:1035] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:40.363063 5366 consensus_queue.cc:260] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.377295 5353 tablet_bootstrap.cc:654] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.372079 5366 ts_tablet_manager.cc:1428] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.094s user 0.005s sys 0.001s
I20250626 19:57:40.383868 5366 tablet_bootstrap.cc:492] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:40.391920 4391 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_f85ad493ebbf4e3b8c7eea1edfcf6a67" } modify_external_catalogs: true
I20250626 19:57:40.392681 4391 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_f85ad493ebbf4e3b8c7eea1edfcf6a67" } modify_external_catalogs: true
I20250626 19:57:40.409584 5410 tablet_bootstrap.cc:654] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.346415 4394 catalog_manager.cc:5582] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 2, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 2 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:40.413816 4394 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:40.417321 4394 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:40.427162 5366 tablet_bootstrap.cc:654] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:40.432081 4385 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e0bcbb044d1549c88e5b83eeb5c709af consensus state has no leader
I20250626 19:57:40.447831 5689 raft_consensus.cc:491] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:40.448277 5689 raft_consensus.cc:513] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.454275 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f834922c9de46dca48c47748913b11a" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:40.456435 5366 tablet_bootstrap.cc:492] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:40.456977 5366 ts_tablet_manager.cc:1397] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.004s
I20250626 19:57:40.457376 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f834922c9de46dca48c47748913b11a" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:40.459270 4611 leader_election.cc:304] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:40.459479 5366 raft_consensus.cc:357] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.460000 5623 raft_consensus.cc:2747] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:40.460230 5366 raft_consensus.cc:383] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.460546 5366 raft_consensus.cc:738] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:40.449992 5689 leader_election.cc:290] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:40.461295 5366 consensus_queue.cc:260] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.470561 5366 ts_tablet_manager.cc:1428] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20250626 19:57:40.477494 5366 tablet_bootstrap.cc:492] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:40.480432 5410 tablet_bootstrap.cc:492] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:40.480825 5410 ts_tablet_manager.cc:1397] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.230s user 0.006s sys 0.004s
I20250626 19:57:40.487219 5366 tablet_bootstrap.cc:654] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.494899 5410 raft_consensus.cc:357] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.495549 5410 raft_consensus.cc:383] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.495784 5410 raft_consensus.cc:738] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:40.496764 5410 consensus_queue.cc:260] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.505743 5410 ts_tablet_manager.cc:1428] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20250626 19:57:40.522902 5642 raft_consensus.cc:491] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:40.523980 5410 tablet_bootstrap.cc:492] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:40.523854 5642 raft_consensus.cc:513] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.526913 5642 leader_election.cc:290] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:40.527153 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "211f6137d9b84afa9aee6432440f7eb8" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:40.528427 4755 leader_election.cc:343] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:40.528816 5433 ts_tablet_manager.cc:1918] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:40.540613 5433 log.cc:1199] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/94327cab01df4338aaec157736d106f6
I20250626 19:57:40.543900 5433 ts_tablet_manager.cc:1939] T 94327cab01df4338aaec157736d106f6 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:40.552183 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 98) took 1399 ms. Trace:
I20250626 19:57:40.567271 5433 rpcz_store.cc:276] 0626 19:57:39.152697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.154883 (+ 2186us) service_pool.cc:225] Handling call
0626 19:57:40.259394 (+1104511us) ts_tablet_manager.cc:1145] Deleting tablet 94327cab01df4338aaec157736d106f6
0626 19:57:40.259432 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:40.474658 (+215226us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:40.528610 (+ 53952us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:40.552122 (+ 23512us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":980439}
I20250626 19:57:40.569334 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 94327cab01df4338aaec157736d106f6 (table default.Table_beec4e8c60ae4e6880c0ff709d3c0ff3 [id=6e84598eef744e8c9b1d918036bc67ec]) successfully deleted
I20250626 19:57:40.569710 5433 tablet_replica.cc:331] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:40.570459 5433 raft_consensus.cc:2241] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:40.571296 5433 pending_rounds.cc:70] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86: Trying to abort 1 pending ops.
I20250626 19:57:40.576788 5433 pending_rounds.cc:77] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964348714475520 op_type: NO_OP noop_request { }
I20250626 19:57:40.578095 5433 raft_consensus.cc:2887] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:57:40.579037 5433 raft_consensus.cc:2270] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:40.588526 5433 ts_tablet_manager.cc:1905] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:40.588526 4395 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 6b8473259485460783f62f8193d5b02c
I20250626 19:57:40.596167 4395 catalog_manager.cc:6652] ReplaceTablet: tablet 6b8473259485460783f62f8193d5b02c deleted and replaced by tablet 8282e22bfb8b4d4994fa8b95ef154a68
I20250626 19:57:40.597046 4395 master_service.cc:948] ReplaceTablet: received request to replace tablet 6b66cda4f8ed410184b434d5083deac4 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:40.561561 5410 tablet_bootstrap.cc:654] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.591037 4648 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b8473259485460783f62f8193d5b02c with delete_type TABLET_DATA_DELETED (replaced by tablet 8282e22bfb8b4d4994fa8b95ef154a68) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:40.599522 4797 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b8473259485460783f62f8193d5b02c with delete_type TABLET_DATA_DELETED (replaced by tablet 8282e22bfb8b4d4994fa8b95ef154a68) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:40.600482 5438 tablet_replica.cc:331] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:40.601233 5438 raft_consensus.cc:2241] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:40.602087 5438 raft_consensus.cc:2270] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:40.604660 5438 ts_tablet_manager.cc:1905] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:40.607965 5353 tablet_bootstrap.cc:492] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:40.608417 5353 ts_tablet_manager.cc:1397] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.641s user 0.008s sys 0.007s
I20250626 19:57:40.611843 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b8473259485460783f62f8193d5b02c with delete_type TABLET_DATA_DELETED (replaced by tablet 8282e22bfb8b4d4994fa8b95ef154a68) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:40.612272 5410 tablet_bootstrap.cc:492] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:40.612655 5410 ts_tablet_manager.cc:1397] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.106s user 0.010s sys 0.002s
I20250626 19:57:40.613248 5653 tablet_replica.cc:331] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:40.614428 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "211f6137d9b84afa9aee6432440f7eb8" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:40.615031 4976 raft_consensus.cc:2466] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:40.615983 4756 leader_election.cc:304] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:40.617384 5506 raft_consensus.cc:2802] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:40.617722 5506 raft_consensus.cc:491] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:40.618018 5506 raft_consensus.cc:3058] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:40.618430 5353 raft_consensus.cc:357] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.619089 5353 raft_consensus.cc:383] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.622747 5353 raft_consensus.cc:738] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:40.623358 5353 consensus_queue.cc:260] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.623878 5410 raft_consensus.cc:357] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.624627 5410 raft_consensus.cc:383] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.628085 5506 raft_consensus.cc:513] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.630695 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "211f6137d9b84afa9aee6432440f7eb8" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:40.631357 4976 raft_consensus.cc:3058] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:40.631770 5506 leader_election.cc:290] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:40.633258 5438 ts_tablet_manager.cc:1918] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:40.633616 5438 log.cc:1199] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/6b8473259485460783f62f8193d5b02c
I20250626 19:57:40.634263 5438 ts_tablet_manager.cc:1939] T 6b8473259485460783f62f8193d5b02c P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:40.634150 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "211f6137d9b84afa9aee6432440f7eb8" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:40.624913 5410 raft_consensus.cc:738] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
W20250626 19:57:40.635484 4755 leader_election.cc:343] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:40.635840 5410 consensus_queue.cc:260] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.641472 5653 raft_consensus.cc:2241] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:40.642931 5653 raft_consensus.cc:2270] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:40.645779 5653 ts_tablet_manager.cc:1905] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:40.649302 5410 ts_tablet_manager.cc:1428] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.036s user 0.002s sys 0.005s
I20250626 19:57:40.650147 5410 tablet_bootstrap.cc:492] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:40.656728 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 6b8473259485460783f62f8193d5b02c (table default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]) successfully deleted
I20250626 19:57:40.662037 5353 ts_tablet_manager.cc:1428] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.053s user 0.005s sys 0.000s
I20250626 19:57:40.671003 5653 ts_tablet_manager.cc:1918] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:40.675722 5653 log.cc:1199] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/6b8473259485460783f62f8193d5b02c
I20250626 19:57:40.676461 5653 ts_tablet_manager.cc:1939] T 6b8473259485460783f62f8193d5b02c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:40.680395 5353 tablet_bootstrap.cc:492] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:40.681222 5410 tablet_bootstrap.cc:654] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.688818 5433 ts_tablet_manager.cc:1918] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:40.689178 5433 log.cc:1199] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/94cefa59de044a36a2afaaf5531128d8
I20250626 19:57:40.689817 5433 ts_tablet_manager.cc:1939] T 94cefa59de044a36a2afaaf5531128d8 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:40.693984 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 6b8473259485460783f62f8193d5b02c (table default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]) successfully deleted
I20250626 19:57:40.712111 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 99) took 1551 ms. Trace:
I20250626 19:57:40.712579 5433 rpcz_store.cc:276] 0626 19:57:39.160287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.160535 (+ 248us) service_pool.cc:225] Handling call
0626 19:57:40.569561 (+1409026us) ts_tablet_manager.cc:1145] Deleting tablet 94cefa59de044a36a2afaaf5531128d8
0626 19:57:40.569605 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:40.632808 (+ 63203us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:40.688607 (+ 55799us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:40.712007 (+ 23400us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290380}
I20250626 19:57:40.714069 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 94cefa59de044a36a2afaaf5531128d8 (table default.Table_c580e8a6ee5d417caa2f7d6811a68349 [id=b8bf68764b2c4a94b7299fa273983e06]) successfully deleted
I20250626 19:57:40.714555 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 103) took 1293 ms. Trace:
I20250626 19:57:40.715008 5433 rpcz_store.cc:276] 0626 19:57:39.421443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.421622 (+ 179us) service_pool.cc:225] Handling call
0626 19:57:40.714217 (+1292595us) ts_tablet_manager.cc:1145] Deleting tablet 1cb6bda529624ac5b5a87a03b4a02d56
0626 19:57:40.714276 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:40.714490 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1291951}
W20250626 19:57:40.716243 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 1cb6bda529624ac5b5a87a03b4a02d56 because the tablet was not found. No further retry: Not found: Tablet not found: 1cb6bda529624ac5b5a87a03b4a02d56
I20250626 19:57:40.725561 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 104) took 1303 ms. Trace:
I20250626 19:57:40.726013 5433 rpcz_store.cc:276] 0626 19:57:39.422560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.422645 (+ 85us) service_pool.cc:225] Handling call
0626 19:57:40.725213 (+1302568us) ts_tablet_manager.cc:1145] Deleting tablet 2ab0320af47049cca3ca87172b9fe5e5
0626 19:57:40.725273 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:40.725493 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1302019}
W20250626 19:57:40.727586 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 2ab0320af47049cca3ca87172b9fe5e5 because the tablet was not found. No further retry: Not found: Tablet not found: 2ab0320af47049cca3ca87172b9fe5e5
I20250626 19:57:40.729969 5410 tablet_bootstrap.cc:492] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:40.730368 5410 ts_tablet_manager.cc:1397] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.080s user 0.019s sys 0.009s
I20250626 19:57:40.731070 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 105) took 1307 ms. Trace:
I20250626 19:57:40.731382 5433 rpcz_store.cc:276] 0626 19:57:39.423463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.472230 (+ 48767us) service_pool.cc:225] Handling call
0626 19:57:40.730827 (+1258597us) ts_tablet_manager.cc:1145] Deleting tablet 4b3bed19e7e34d24b995840a3bde5ef0
0626 19:57:40.730866 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:40.731021 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1258090}
W20250626 19:57:40.732368 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 4b3bed19e7e34d24b995840a3bde5ef0 because the tablet was not found. No further retry: Not found: Tablet not found: 4b3bed19e7e34d24b995840a3bde5ef0
I20250626 19:57:40.732852 5410 raft_consensus.cc:357] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.733475 5410 raft_consensus.cc:383] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.733712 5410 raft_consensus.cc:738] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:40.734300 5410 consensus_queue.cc:260] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.737517 5410 ts_tablet_manager.cc:1428] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250626 19:57:40.738299 5410 tablet_bootstrap.cc:492] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:40.744168 5410 tablet_bootstrap.cc:654] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.732383 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 109) took 1138 ms. Trace:
I20250626 19:57:40.745961 5433 rpcz_store.cc:276] 0626 19:57:39.594606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.624250 (+ 29644us) service_pool.cc:225] Handling call
0626 19:57:40.732168 (+1107918us) ts_tablet_manager.cc:1145] Deleting tablet a31f1a7739544706bff2a500d6026086
0626 19:57:40.732214 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:40.732341 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1096716}
I20250626 19:57:40.747092 5433 tablet_replica.cc:331] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
W20250626 19:57:40.747397 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet a31f1a7739544706bff2a500d6026086 because the tablet was not found. No further retry: Not found: Tablet not found: a31f1a7739544706bff2a500d6026086
I20250626 19:57:40.747746 5433 raft_consensus.cc:2241] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:40.748210 5433 raft_consensus.cc:2270] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:40.750244 5433 ts_tablet_manager.cc:1905] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:40.757033 5410 tablet_bootstrap.cc:492] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:40.757498 5410 ts_tablet_manager.cc:1397] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.000s
I20250626 19:57:40.759524 5410 raft_consensus.cc:357] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.760233 5410 raft_consensus.cc:383] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.760524 5410 raft_consensus.cc:738] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:40.761226 5410 consensus_queue.cc:260] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.764675 5410 ts_tablet_manager.cc:1428] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250626 19:57:40.765583 5410 tablet_bootstrap.cc:492] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
W20250626 19:57:31.812295 5387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:57:31.812772 5266 server_base.cc:1048] running on GCE node
W20250626 19:57:31.812120 5382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:57:40.771797 5266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 19:57:40.774158 4976 raft_consensus.cc:2466] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:40.779877 5410 tablet_bootstrap.cc:654] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.787501 5266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 19:57:40.790227 5410 tablet_bootstrap.cc:492] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:40.790681 5410 ts_tablet_manager.cc:1397] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.000s
I20250626 19:57:40.792963 5410 raft_consensus.cc:357] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.794718 5410 raft_consensus.cc:383] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.795235 5266 hybrid_clock.cc:648] HybridClock initialized: now 1750967860795142 us; error 93 us; skew 500 ppm
I20250626 19:57:40.796316 5266 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 19:57:40.797930 4391 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet f807cf54bbb146ff989762c18410e6bb
I20250626 19:57:40.795063 5410 raft_consensus.cc:738] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:40.799782 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:40.800437 5410 consensus_queue.cc:260] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:40.801538 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet f807cf54bbb146ff989762c18410e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f807cf54bbb146ff989762c18410e6bb already in progress: creating tablet
I20250626 19:57:40.801934 5433 ts_tablet_manager.cc:1918] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:40.802266 5433 log.cc:1199] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/82823fd9ceb1485386616319fa7aeb43
I20250626 19:57:40.802870 5433 ts_tablet_manager.cc:1939] T 82823fd9ceb1485386616319fa7aeb43 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:40.803535 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 110) took 1149 ms. Trace:
I20250626 19:57:40.803812 5433 rpcz_store.cc:276] 0626 19:57:39.656428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.656675 (+ 247us) service_pool.cc:225] Handling call
0626 19:57:40.746960 (+1090285us) ts_tablet_manager.cc:1145] Deleting tablet 82823fd9ceb1485386616319fa7aeb43
0626 19:57:40.746997 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:40.770326 (+ 23329us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:40.801729 (+ 31403us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:40.803491 (+ 1762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1089794}
I20250626 19:57:40.804934 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 111) took 1147 ms. Trace:
I20250626 19:57:40.805351 4648 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:40.805977 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 82823fd9ceb1485386616319fa7aeb43 (table default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]) successfully deleted
I20250626 19:57:40.807631 4797 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:40742
W20250626 19:57:40.809167 4383 catalog_manager.cc:4908] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet f807cf54bbb146ff989762c18410e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f807cf54bbb146ff989762c18410e6bb already in progress: creating tablet
I20250626 19:57:40.817065 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_f26100f290884036a30e2b11e0826db6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:40.821019 5410 ts_tablet_manager.cc:1428] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s
I20250626 19:57:40.805202 5433 rpcz_store.cc:276] 0626 19:57:39.657473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.659972 (+ 2499us) service_pool.cc:225] Handling call
0626 19:57:40.804724 (+1144752us) ts_tablet_manager.cc:1145] Deleting tablet fd6543699e254400b3516aece4486fe0
0626 19:57:40.804760 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:40.804891 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1144264}
I20250626 19:57:40.823694 5410 tablet_bootstrap.cc:492] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
W20250626 19:57:40.823721 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet fd6543699e254400b3516aece4486fe0 because the tablet was not found. No further retry: Not found: Tablet not found: fd6543699e254400b3516aece4486fe0
I20250626 19:57:40.825740 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 112) took 1165 ms. Trace:
I20250626 19:57:40.826151 5433 rpcz_store.cc:276] 0626 19:57:39.660787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.661207 (+ 420us) service_pool.cc:225] Handling call
0626 19:57:40.825426 (+1164219us) ts_tablet_manager.cc:1145] Deleting tablet c92deaa237064dee8f866e3a7a355ca4
0626 19:57:40.825484 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:40.825674 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1163725}
W20250626 19:57:40.819494 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f26100f290884036a30e2b11e0826db6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:57:40.827510 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet c92deaa237064dee8f866e3a7a355ca4 because the tablet was not found. No further retry: Not found: Tablet not found: c92deaa237064dee8f866e3a7a355ca4
I20250626 19:57:40.828508 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 113) took 1166 ms. Trace:
I20250626 19:57:40.828922 5433 rpcz_store.cc:276] 0626 19:57:39.662032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.664742 (+ 2710us) service_pool.cc:225] Handling call
0626 19:57:40.828218 (+1163476us) ts_tablet_manager.cc:1145] Deleting tablet 36997dab0ab3430bba63a90805759b30
0626 19:57:40.828253 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:40.828459 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1162971}
W20250626 19:57:40.830077 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 36997dab0ab3430bba63a90805759b30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36997dab0ab3430bba63a90805759b30 already in progress: creating tablet
I20250626 19:57:40.831068 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 114) took 1165 ms. Trace:
I20250626 19:57:40.834882 5433 rpcz_store.cc:276] 0626 19:57:39.665571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.666834 (+ 1263us) service_pool.cc:225] Handling call
0626 19:57:40.830785 (+1163951us) ts_tablet_manager.cc:1145] Deleting tablet cb34ffcd4da04e32a1220b3f298ec22d
0626 19:57:40.830819 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:40.831024 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1163372}
W20250626 19:57:40.836494 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet cb34ffcd4da04e32a1220b3f298ec22d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb34ffcd4da04e32a1220b3f298ec22d already in progress: creating tablet
I20250626 19:57:40.837535 5433 tablet_replica.cc:331] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:40.838351 5433 raft_consensus.cc:2241] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:40.838919 5433 raft_consensus.cc:2270] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:40.843946 5410 tablet_bootstrap.cc:654] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.847362 5506 raft_consensus.cc:491] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:40.847832 5506 raft_consensus.cc:513] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.849907 5433 ts_tablet_manager.cc:1905] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:40.850996 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "465ed0fe56fb4960a58e92b9b3d594e6" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:40.851696 5266 webserver.cc:469] Webserver started at http://127.4.46.190:36587/ using document root <none> and password file <none>
I20250626 19:57:40.853235 5266 fs_manager.cc:362] Metadata directory not provided
I20250626 19:57:40.853225 5506 leader_election.cc:290] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:40.853890 5266 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 19:57:40.856448 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:40.852249 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "465ed0fe56fb4960a58e92b9b3d594e6" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:40.859949 4755 leader_election.cc:343] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:40.869406 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:40.873445 5410 tablet_bootstrap.cc:492] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:40.873845 5410 ts_tablet_manager.cc:1397] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.004s
I20250626 19:57:40.876960 5410 raft_consensus.cc:357] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.877594 5410 raft_consensus.cc:383] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.878062 5410 raft_consensus.cc:738] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:40.878741 5410 consensus_queue.cc:260] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.882170 5410 ts_tablet_manager.cc:1428] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250626 19:57:40.883381 5410 tablet_bootstrap.cc:492] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:40.888198 5353 tablet_bootstrap.cc:654] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.891016 5623 raft_consensus.cc:491] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:40.891469 5623 raft_consensus.cc:513] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.893883 5623 leader_election.cc:290] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:40.895529 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c86ced5a3dc48e380244a9548d2f4c8" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:40.897755 5410 tablet_bootstrap.cc:654] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.898967 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c86ced5a3dc48e380244a9548d2f4c8" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:40.906790 5410 tablet_bootstrap.cc:492] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:40.907186 5410 ts_tablet_manager.cc:1397] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.004s
I20250626 19:57:40.907922 4611 leader_election.cc:304] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:40.909921 5623 raft_consensus.cc:2747] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:40.912835 5410 raft_consensus.cc:357] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.913439 5410 raft_consensus.cc:383] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.913694 5410 raft_consensus.cc:738] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:40.914214 5410 consensus_queue.cc:260] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.915755 5410 ts_tablet_manager.cc:1428] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250626 19:57:40.916337 4756 leader_election.cc:304] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:40.916574 5410 tablet_bootstrap.cc:492] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:40.917111 5642 raft_consensus.cc:2802] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:40.917531 5642 raft_consensus.cc:695] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:40.918285 5642 consensus_queue.cc:237] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:40.926092 4756 leader_election.cc:343] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:40.926960 4756 leader_election.cc:304] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:40.927842 5688 raft_consensus.cc:2747] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:40.962896 5366 tablet_bootstrap.cc:492] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:40.954033 4675 raft_consensus.cc:3058] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:40.963310 5366 ts_tablet_manager.cc:1397] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.492s user 0.108s sys 0.106s
I20250626 19:57:40.965557 5366 raft_consensus.cc:357] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.965997 4396 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:40.966148 5366 raft_consensus.cc:383] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.966372 5366 raft_consensus.cc:738] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:40.967147 5366 consensus_queue.cc:260] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.969096 5366 ts_tablet_manager.cc:1428] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250626 19:57:40.969892 5410 tablet_bootstrap.cc:654] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.972946 5366 tablet_bootstrap.cc:492] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:40.922986 5433 ts_tablet_manager.cc:1918] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:40.977864 5433 log.cc:1199] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/dd0ca2fab40f45e5a1f9bd6059ca1f08
I20250626 19:57:40.978229 5366 tablet_bootstrap.cc:654] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:40.978896 5433 ts_tablet_manager.cc:1939] T dd0ca2fab40f45e5a1f9bd6059ca1f08 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:40.982925 5410 tablet_bootstrap.cc:492] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:40.984570 5410 ts_tablet_manager.cc:1397] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.026s
I20250626 19:57:40.986881 5410 raft_consensus.cc:357] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.987648 5410 raft_consensus.cc:383] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.987887 5410 raft_consensus.cc:738] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:40.989748 5366 tablet_bootstrap.cc:492] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:40.990123 5366 ts_tablet_manager.cc:1397] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.003s
I20250626 19:57:40.992805 5410 consensus_queue.cc:260] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.994697 5410 ts_tablet_manager.cc:1428] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250626 19:57:40.994978 5366 raft_consensus.cc:357] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.995564 5366 raft_consensus.cc:383] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:40.995803 5366 raft_consensus.cc:738] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:40.996312 5366 consensus_queue.cc:260] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:40.997879 5366 ts_tablet_manager.cc:1428] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20250626 19:57:40.998705 5366 tablet_bootstrap.cc:492] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:41.004318 5366 tablet_bootstrap.cc:654] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:41.015802 5410 tablet_bootstrap.cc:492] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:41.016484 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 118) took 1194 ms. Trace:
I20250626 19:57:41.016798 5433 rpcz_store.cc:276] 0626 19:57:39.822423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.822519 (+ 96us) service_pool.cc:225] Handling call
0626 19:57:40.837403 (+1014884us) ts_tablet_manager.cc:1145] Deleting tablet dd0ca2fab40f45e5a1f9bd6059ca1f08
0626 19:57:40.837443 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:40.888066 (+ 50623us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:40.922687 (+ 34621us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:41.016426 (+ 93739us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1014320}
I20250626 19:57:41.017906 5433 tablet_replica.cc:331] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:41.019822 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 (table default.Table_889f18064aab4774b17bfa478a0a7e33 [id=0df063bc889246c3ba841b1b2f8e6104]) successfully deleted
I20250626 19:57:41.023382 5410 tablet_bootstrap.cc:654] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:41.028210 5366 tablet_bootstrap.cc:492] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:41.028756 5366 ts_tablet_manager.cc:1397] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.007s
I20250626 19:57:41.032552 5433 raft_consensus.cc:2241] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:41.032404 5366 raft_consensus.cc:357] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.033041 5433 raft_consensus.cc:2270] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:41.033245 5366 raft_consensus.cc:383] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:41.035288 5433 ts_tablet_manager.cc:1905] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:41.037262 5366 raft_consensus.cc:738] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:41.037994 5366 consensus_queue.cc:260] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.039948 5366 ts_tablet_manager.cc:1428] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250626 19:57:41.041498 5366 tablet_bootstrap.cc:492] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:41.047755 5410 tablet_bootstrap.cc:492] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:41.048226 5410 ts_tablet_manager.cc:1397] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.002s
I20250626 19:57:41.050622 5410 raft_consensus.cc:357] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.055701 5366 tablet_bootstrap.cc:654] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:41.056010 5410 raft_consensus.cc:383] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:41.056336 5410 raft_consensus.cc:738] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:41.056994 5410 consensus_queue.cc:260] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.059582 5410 ts_tablet_manager.cc:1428] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.011s user 0.003s sys 0.001s
I20250626 19:57:41.060462 5410 tablet_bootstrap.cc:492] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:41.064211 5353 tablet_bootstrap.cc:492] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:41.064612 5353 ts_tablet_manager.cc:1397] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.384s user 0.010s sys 0.000s
I20250626 19:57:41.069187 5410 tablet_bootstrap.cc:654] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:41.072904 5266 fs_manager.cc:714] Time spent opening directory manager: real 0.214s user 0.053s sys 0.076s
I20250626 19:57:41.074359 4396 catalog_manager.cc:5582] T e60b4d53c40c4dffab928f0e3fb4a80b P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:41.081231 5689 raft_consensus.cc:491] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.081691 5689 raft_consensus.cc:513] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.083396 5689 leader_election.cc:290] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.084477 5768 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 19:57:41.084853 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c113e03594b428a979435618fe373c6" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:41.086186 4611 leader_election.cc:343] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:41.086720 5266 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.003s sys 0.001s
I20250626 19:57:41.087122 5266 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data,/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/wal
uuid: "e6c58006b6be41c1bd6bdb5eb010d360"
format_stamp: "Formatted at 2025-06-26 19:57:12 on dist-test-slave-48dm"
I20250626 19:57:41.090569 5266 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 19:57:41.095003 5353 raft_consensus.cc:357] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.095886 5353 raft_consensus.cc:383] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:41.096187 5353 raft_consensus.cc:738] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:41.097031 5353 consensus_queue.cc:260] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.115329 4675 raft_consensus.cc:3058] T 6c86ced5a3dc48e380244a9548d2f4c8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.136379 4396 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 6b66cda4f8ed410184b434d5083deac4 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]). Expected version 1 got 0
I20250626 19:57:41.148509 5353 ts_tablet_manager.cc:1428] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.084s user 0.007s sys 0.001s
I20250626 19:57:41.150712 5353 tablet_bootstrap.cc:492] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:41.164364 5410 tablet_bootstrap.cc:492] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:41.164777 5410 ts_tablet_manager.cc:1397] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.104s user 0.030s sys 0.035s
I20250626 19:57:41.166930 5410 raft_consensus.cc:357] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.167528 5410 raft_consensus.cc:383] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:41.167785 5410 raft_consensus.cc:738] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:41.191922 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c113e03594b428a979435618fe373c6" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:41.193403 4819 raft_consensus.cc:2466] T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:41.197814 5623 raft_consensus.cc:491] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.198426 5623 raft_consensus.cc:513] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.200735 5623 leader_election.cc:290] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.202478 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5706841a14bd4ece965098b88ccb7ec8" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:41.203238 5410 consensus_queue.cc:260] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:41.203858 4611 leader_election.cc:343] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:41.204659 5623 raft_consensus.cc:491] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.205377 4396 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:41.205900 5623 raft_consensus.cc:513] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.206844 4396 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:41.207363 4396 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:41.208671 4396 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:41.209790 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6cc7a336385f499ca19566d5f2901e98" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:41.212746 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5706841a14bd4ece965098b88ccb7ec8" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:41.213439 4820 raft_consensus.cc:2466] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:41.216778 4609 leader_election.cc:304] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:41.217484 5689 raft_consensus.cc:2802] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:41.218001 5689 raft_consensus.cc:491] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:41.218429 5689 raft_consensus.cc:3058] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.213629 5623 leader_election.cc:290] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.229952 5410 ts_tablet_manager.cc:1428] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.065s user 0.006s sys 0.001s
I20250626 19:57:41.231048 5410 tablet_bootstrap.cc:492] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:41.236387 5774 raft_consensus.cc:491] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.237295 5774 raft_consensus.cc:513] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.240351 5774 leader_election.cc:290] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:41.250689 5623 raft_consensus.cc:491] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.251366 5623 raft_consensus.cc:513] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.254360 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ba78c1b40ef4925b4108e1455bcfc44" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:41.255774 4611 leader_election.cc:343] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:41.257797 5623 leader_election.cc:290] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.227391 5433 ts_tablet_manager.cc:1918] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:41.263370 5433 log.cc:1199] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/f43aed8f07844c5ca727ae6df63896df
I20250626 19:57:41.264307 5433 ts_tablet_manager.cc:1939] T f43aed8f07844c5ca727ae6df63896df P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:41.261085 5506 raft_consensus.cc:491] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.267082 5506 raft_consensus.cc:513] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.269064 5506 leader_election.cc:290] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:41.264634 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:41.269543 5366 tablet_bootstrap.cc:492] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:41.270282 5366 ts_tablet_manager.cc:1397] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.229s user 0.010s sys 0.013s
I20250626 19:57:41.217999 4609 leader_election.cc:304] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
W20250626 19:57:41.279969 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 62057.2us
I20250626 19:57:41.280459 5775 raft_consensus.cc:2802] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:41.280805 5775 raft_consensus.cc:491] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:41.281098 5775 raft_consensus.cc:3058] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.281307 4675 raft_consensus.cc:3058] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.282084 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6cc7a336385f499ca19566d5f2901e98" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:41.282785 4820 raft_consensus.cc:2391] T 6cc7a336385f499ca19566d5f2901e98 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:41.285964 5366 raft_consensus.cc:357] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.287797 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 119) took 1464 ms. Trace:
I20250626 19:57:41.288121 5433 rpcz_store.cc:276] 0626 19:57:39.823406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.823493 (+ 87us) service_pool.cc:225] Handling call
0626 19:57:41.017768 (+1194275us) ts_tablet_manager.cc:1145] Deleting tablet f43aed8f07844c5ca727ae6df63896df
0626 19:57:41.017805 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:41.060218 (+ 42413us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:41.227167 (+166949us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:41.287741 (+ 60574us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13796,"tablet-delete.queue_time_us":1193758}
I20250626 19:57:41.289479 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet f43aed8f07844c5ca727ae6df63896df (table default.Table_d6eed56e9d514833bfe29ff3aba9718a [id=07c980b0511a43c0aa406597788cd05b]) successfully deleted
I20250626 19:57:41.261087 5689 raft_consensus.cc:513] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.292798 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c113e03594b428a979435618fe373c6" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
W20250626 19:57:41.294140 4611 leader_election.cc:343] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:41.295003 5689 leader_election.cc:290] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.297462 5353 tablet_bootstrap.cc:654] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:41.290572 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 123) took 1241 ms. Trace:
I20250626 19:57:41.300457 5433 rpcz_store.cc:276] 0626 19:57:40.049317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:40.049455 (+ 138us) service_pool.cc:225] Handling call
0626 19:57:41.290247 (+1240792us) ts_tablet_manager.cc:1145] Deleting tablet fc526a05990446a3b3bc973e79511476
0626 19:57:41.290287 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:41.290525 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1240144}
W20250626 19:57:41.309679 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet fc526a05990446a3b3bc973e79511476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc526a05990446a3b3bc973e79511476 already in progress: creating tablet
I20250626 19:57:41.329032 5266 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 19:57:41.331001 5266 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 19:57:41.331557 5266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 19:57:41.333818 5410 tablet_bootstrap.cc:654] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:41.335531 5664 consensus_queue.cc:1035] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:41.338474 4609 leader_election.cc:304] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:41.348522 5623 raft_consensus.cc:2747] T 6cc7a336385f499ca19566d5f2901e98 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:41.349164 5623 raft_consensus.cc:491] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.349606 5623 raft_consensus.cc:513] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.352357 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "211f6137d9b84afa9aee6432440f7eb8" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:41.353073 4976 raft_consensus.cc:2391] T 211f6137d9b84afa9aee6432440f7eb8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 93f885c6802341c889a40e9c20fbfff1 in this term.
I20250626 19:57:41.354612 5623 leader_election.cc:290] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.360910 4395 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 6b66cda4f8ed410184b434d5083deac4
I20250626 19:57:41.363270 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b66cda4f8ed410184b434d5083deac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7851e464fcd44ecb82404d6a991db2a2) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:41.286590 5366 raft_consensus.cc:383] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:41.364755 5366 raft_consensus.cc:738] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:41.366192 4395 catalog_manager.cc:6652] ReplaceTablet: tablet 6b66cda4f8ed410184b434d5083deac4 deleted and replaced by tablet 7851e464fcd44ecb82404d6a991db2a2
I20250626 19:57:41.366429 4648 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b66cda4f8ed410184b434d5083deac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7851e464fcd44ecb82404d6a991db2a2) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:41.367169 5433 tablet_replica.cc:331] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:41.367897 5433 raft_consensus.cc:2241] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:41.368443 5433 raft_consensus.cc:2270] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:41.367060 4395 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34796 (request call id 140) took 1092 ms. Trace:
I20250626 19:57:41.376700 4797 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b66cda4f8ed410184b434d5083deac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7851e464fcd44ecb82404d6a991db2a2) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:41.377833 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c113e03594b428a979435618fe373c6" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:41.378533 4817 raft_consensus.cc:3058] T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.383419 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ba78c1b40ef4925b4108e1455bcfc44" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:41.369004 4395 rpcz_store.cc:276] 0626 19:57:40.274560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:40.596927 (+322367us) service_pool.cc:225] Handling call
0626 19:57:40.711057 (+114130us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe791559891 00007fe78f2943a7 00007fe78f294332 00007fe78f294261 00007fe78f2941dc 00007fe78f293492 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:41.366776 (+655719us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:41.343848 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:41.344236 (+ 388us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:41.344267 (+ 31us) write_op.cc:435] Acquired schema lock
0626 19:57:41.344284 (+ 17us) tablet.cc:660] Decoding operations
0626 19:57:41.344653 (+ 369us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:41.344695 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:41.344716 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:41.344934 (+ 218us) tablet.cc:699] Row locks acquired
0626 19:57:41.344955 (+ 21us) write_op.cc:260] PREPARE: finished
0626 19:57:41.345105 (+ 150us) write_op.cc:270] Start()
0626 19:57:41.345226 (+ 121us) write_op.cc:276] Timestamp: P: 1750967861345169 usec, L: 0
0626 19:57:41.345249 (+ 23us) op_driver.cc:348] REPLICATION: starting
0626 19:57:41.345885 (+ 636us) log.cc:844] Serialized 659 byte log entry
0626 19:57:41.352724 (+ 6839us) op_driver.cc:464] REPLICATION: finished
0626 19:57:41.353115 (+ 391us) write_op.cc:301] APPLY: starting
0626 19:57:41.353167 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:41.353273 (+ 106us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:41.353366 (+ 93us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:41.353682 (+ 316us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:41.353912 (+ 230us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:41.353955 (+ 43us) write_op.cc:312] APPLY: finished
0626 19:57:41.354134 (+ 179us) log.cc:844] Serialized 29 byte log entry
0626 19:57:41.354269 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:41.354339 (+ 70us) write_op.cc:454] Released schema lock
0626 19:57:41.354369 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:41.354616 (+ 247us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:41.354644 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":259394816,"child_traces":[["op",{"apply.queue_time_us":244,"apply.run_cpu_time_us":1455,"apply.run_wall_time_us":1454,"num_ops":2,"prepare.queue_time_us":2101,"prepare.run_cpu_time_us":2386,"prepare.run_wall_time_us":2384,"raft.queue_time_us":1148,"raft.run_cpu_time_us":680,"raft.run_wall_time_us":679,"replication_time_us":7380}]]}
I20250626 19:57:41.385660 4820 raft_consensus.cc:2466] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:41.395406 4817 raft_consensus.cc:2466] T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:41.403105 4675 raft_consensus.cc:3058] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.406378 5366 consensus_queue.cc:260] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.416919 5366 ts_tablet_manager.cc:1428] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.146s user 0.006s sys 0.000s
I20250626 19:57:41.421487 5433 ts_tablet_manager.cc:1905] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:41.431854 5782 tablet_replica.cc:331] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:41.452402 5773 raft_consensus.cc:491] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.452831 5773 raft_consensus.cc:513] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.454998 5773 leader_election.cc:290] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:41.451098 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "211f6137d9b84afa9aee6432440f7eb8" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:41.425261 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1474bfa74a6346bf9e3ff49019f6c9b3" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:41.427325 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e65b16141c3d47c5b66024da59b9bbe9" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:41.476094 4385 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e0bcbb044d1549c88e5b83eeb5c709af consensus state has no leader
W20250626 19:57:41.485105 4755 leader_election.cc:343] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:41.485913 4755 leader_election.cc:343] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:41.436657 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1474bfa74a6346bf9e3ff49019f6c9b3" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:41.449398 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e65b16141c3d47c5b66024da59b9bbe9" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:41.472925 5664 raft_consensus.cc:491] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.491576 5366 tablet_bootstrap.cc:492] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:41.491601 5664 raft_consensus.cc:513] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.480933 5353 tablet_bootstrap.cc:492] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:41.497786 5353 ts_tablet_manager.cc:1397] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.347s user 0.006s sys 0.007s
I20250626 19:57:41.500787 5664 leader_election.cc:290] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.501256 5664 raft_consensus.cc:491] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.501655 5664 raft_consensus.cc:513] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.506914 5782 raft_consensus.cc:2241] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:41.507524 5782 raft_consensus.cc:2270] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:41.437319 4609 leader_election.cc:304] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:41.509752 5623 raft_consensus.cc:2802] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250626 19:57:41.509893 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 72679.5us
I20250626 19:57:41.510161 5623 raft_consensus.cc:491] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:41.510578 5623 raft_consensus.cc:3058] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.510530 4609 leader_election.cc:304] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:41.511276 5689 raft_consensus.cc:2802] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:41.511610 4609 leader_election.cc:304] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
W20250626 19:57:41.514874 5792 log.cc:927] Time spent T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20250626 19:57:41.528874 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0bcbb044d1549c88e5b83eeb5c709af" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:41.529557 4820 raft_consensus.cc:2466] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:41.531205 5785 tablet_replica.cc:331] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:41.485126 5775 raft_consensus.cc:513] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.533421 5785 raft_consensus.cc:2241] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:41.484243 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3636b78f4a64f9eacb4e5979768929b" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:41.531340 5801 raft_consensus.cc:3058] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.531716 5689 raft_consensus.cc:695] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:41.533874 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5706841a14bd4ece965098b88ccb7ec8" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:41.534308 5785 raft_consensus.cc:2270] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:41.534425 5689 consensus_queue.cc:237] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:41.534884 4755 leader_election.cc:343] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:41.535145 4611 leader_election.cc:343] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:41.536003 5775 leader_election.cc:290] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.536535 5785 ts_tablet_manager.cc:1905] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:41.537642 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3636b78f4a64f9eacb4e5979768929b" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:41.547540 5353 raft_consensus.cc:357] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.548382 5353 raft_consensus.cc:383] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:41.548672 5353 raft_consensus.cc:738] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:41.549368 5353 consensus_queue.cc:260] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.551189 5664 leader_election.cc:290] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.563261 5801 raft_consensus.cc:2747] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:41.569633 5410 tablet_bootstrap.cc:492] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:41.570107 5410 ts_tablet_manager.cc:1397] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.339s user 0.009s sys 0.005s
I20250626 19:57:41.570117 5353 ts_tablet_manager.cc:1428] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.072s user 0.006s sys 0.000s
I20250626 19:57:41.577456 5782 ts_tablet_manager.cc:1905] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:41.581141 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:41.582185 5353 tablet_bootstrap.cc:492] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:41.522275 5623 raft_consensus.cc:513] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.585812 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ba78c1b40ef4925b4108e1455bcfc44" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
W20250626 19:57:41.587121 4611 leader_election.cc:343] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:41.583427 4399 catalog_manager.cc:5582] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:41.591825 4911 leader_election.cc:304] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:41.593590 5664 raft_consensus.cc:2802] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:41.593991 5664 raft_consensus.cc:491] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:41.594360 5664 raft_consensus.cc:3058] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.604051 4399 catalog_manager.cc:5582] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:41.586921 5410 raft_consensus.cc:357] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.601812 5689 raft_consensus.cc:491] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.607802 5689 raft_consensus.cc:513] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.609853 5366 tablet_bootstrap.cc:654] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:41.610332 5689 leader_election.cc:290] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.612159 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8122b26848141a39d2462dd3491f9cd" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:41.612934 4979 raft_consensus.cc:2466] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:41.615554 4399 catalog_manager.cc:5582] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:41.614351 4611 leader_election.cc:304] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:41.619530 5689 raft_consensus.cc:2802] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:41.619881 5689 raft_consensus.cc:491] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:41.620182 5689 raft_consensus.cc:3058] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:41.622823 4756 leader_election.cc:343] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:41.623214 4756 leader_election.cc:304] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
W20250626 19:57:41.611855 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 100361us
W20250626 19:57:41.624713 4756 leader_election.cc:343] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:41.625072 4756 leader_election.cc:304] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:41.625500 5353 tablet_bootstrap.cc:654] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:41.623975 5772 raft_consensus.cc:2747] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:41.627336 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5706841a14bd4ece965098b88ccb7ec8" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:41.627827 4820 raft_consensus.cc:3058] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.631222 5801 raft_consensus.cc:491] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.631743 5664 raft_consensus.cc:513] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.631734 5801 raft_consensus.cc:513] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.634096 5801 leader_election.cc:290] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.635195 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8122b26848141a39d2462dd3491f9cd" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:41.635789 4817 raft_consensus.cc:2466] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:41.634539 4820 raft_consensus.cc:2466] T 5706841a14bd4ece965098b88ccb7ec8 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:41.625924 5802 raft_consensus.cc:491] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.637001 5802 raft_consensus.cc:513] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.635804 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0bcbb044d1549c88e5b83eeb5c709af" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:41.638310 4979 raft_consensus.cc:2391] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:41.638306 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:41.639309 5802 leader_election.cc:290] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.640043 5772 raft_consensus.cc:2747] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:41.641551 4756 leader_election.cc:343] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:41.642015 4756 leader_election.cc:304] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:41.642712 5772 raft_consensus.cc:2747] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:41.642680 4611 leader_election.cc:343] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Not found: Tablet not found: 36997dab0ab3430bba63a90805759b30
I20250626 19:57:41.645285 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8122b26848141a39d2462dd3491f9cd" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:41.645937 4820 raft_consensus.cc:2466] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:41.647017 5664 leader_election.cc:290] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.652889 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0bcbb044d1549c88e5b83eeb5c709af" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:41.653908 4670 raft_consensus.cc:2466] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:41.655236 5623 leader_election.cc:290] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.638911 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:41.656132 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8122b26848141a39d2462dd3491f9cd" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:41.656466 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:41.659623 5410 raft_consensus.cc:383] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:41.661710 5410 raft_consensus.cc:738] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:41.663918 5410 consensus_queue.cc:260] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.664980 5353 tablet_bootstrap.cc:492] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:41.665408 5353 ts_tablet_manager.cc:1397] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.004s
I20250626 19:57:41.667834 5353 raft_consensus.cc:357] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.668462 5353 raft_consensus.cc:383] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:41.668692 5353 raft_consensus.cc:738] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:41.669217 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0bcbb044d1549c88e5b83eeb5c709af" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:41.669715 4817 raft_consensus.cc:3058] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.671470 5785 ts_tablet_manager.cc:1918] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:57:41.671861 5785 log.cc:1199] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/6b66cda4f8ed410184b434d5083deac4
I20250626 19:57:41.675486 5353 consensus_queue.cc:260] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.674885 5785 ts_tablet_manager.cc:1939] T 6b66cda4f8ed410184b434d5083deac4 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:41.676906 4817 raft_consensus.cc:2466] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:41.664925 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0bcbb044d1549c88e5b83eeb5c709af" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:41.682104 4675 raft_consensus.cc:3058] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.685009 5773 raft_consensus.cc:491] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.685389 5773 raft_consensus.cc:513] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.687127 5773 leader_election.cc:290] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:41.688764 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6740fcfd3d41487b8a4cb2a6af1e70d4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:41.690184 5366 tablet_bootstrap.cc:492] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:41.690557 5366 ts_tablet_manager.cc:1397] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.199s user 0.008s sys 0.002s
W20250626 19:57:41.690490 4755 leader_election.cc:343] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:41.697497 5772 consensus_queue.cc:1035] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:41.700280 5410 ts_tablet_manager.cc:1428] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.130s user 0.007s sys 0.001s
I20250626 19:57:41.702422 5410 tablet_bootstrap.cc:492] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:41.707638 5366 raft_consensus.cc:357] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:41.708333 5366 raft_consensus.cc:383] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:41.708590 5366 raft_consensus.cc:738] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:41.711280 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ba78c1b40ef4925b4108e1455bcfc44" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:41.712030 4820 raft_consensus.cc:3058] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.721962 4609 leader_election.cc:304] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:41.723691 5623 raft_consensus.cc:2802] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:41.724365 5623 raft_consensus.cc:695] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:41.725529 5623 consensus_queue.cc:237] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.726632 5366 consensus_queue.cc:260] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:41.727627 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 6b66cda4f8ed410184b434d5083deac4 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]) successfully deleted
I20250626 19:57:41.679749 4911 leader_election.cc:304] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
W20250626 19:57:41.729424 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 49793.2us
I20250626 19:57:41.730341 4911 leader_election.cc:304] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:41.693949 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6740fcfd3d41487b8a4cb2a6af1e70d4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:41.707855 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0bcbb044d1549c88e5b83eeb5c709af" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:41.731423 5807 raft_consensus.cc:2802] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:41.731571 4817 raft_consensus.cc:2391] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:41.731739 5807 raft_consensus.cc:491] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:41.732020 5807 raft_consensus.cc:3058] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.733742 5433 ts_tablet_manager.cc:1918] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:41.734170 5433 log.cc:1199] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/6b8473259485460783f62f8193d5b02c
I20250626 19:57:41.734931 5433 ts_tablet_manager.cc:1939] T 6b8473259485460783f62f8193d5b02c P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:41.735031 5794 consensus_queue.cc:1035] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:41.735673 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 124) took 1145 ms. Trace:
I20250626 19:57:41.736217 5794 raft_consensus.cc:2802] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:41.735955 5433 rpcz_store.cc:276] 0626 19:57:40.590671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:40.590846 (+ 175us) service_pool.cc:225] Handling call
0626 19:57:41.310729 (+719883us) ts_tablet_manager.cc:1145] Deleting tablet 6b8473259485460783f62f8193d5b02c
0626 19:57:41.367007 (+ 56278us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:41.678774 (+311767us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:41.733512 (+ 54738us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:41.735613 (+ 2101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":711374}
I20250626 19:57:41.736598 5794 raft_consensus.cc:695] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:41.737571 5433 tablet_replica.cc:331] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:41.737298 5794 consensus_queue.cc:237] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.697901 4675 raft_consensus.cc:2466] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:41.738313 5433 raft_consensus.cc:2241] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:41.740258 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 6b8473259485460783f62f8193d5b02c (table default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]) successfully deleted
W20250626 19:57:41.740902 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet f807cf54bbb146ff989762c18410e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f807cf54bbb146ff989762c18410e6bb already in progress: creating tablet
I20250626 19:57:41.741041 5366 ts_tablet_manager.cc:1428] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.050s user 0.005s sys 0.000s
I20250626 19:57:41.741870 5366 tablet_bootstrap.cc:492] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:41.705854 5689 raft_consensus.cc:513] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.744069 4670 raft_consensus.cc:2391] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:41.749486 5689 leader_election.cc:290] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.769594 4820 raft_consensus.cc:2466] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
W20250626 19:57:41.759563 4609 leader_election.cc:343] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Not found: Tablet not found: 36997dab0ab3430bba63a90805759b30
I20250626 19:57:41.770673 4609 leader_election.cc:304] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:41.772221 4609 leader_election.cc:304] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:41.773131 5802 raft_consensus.cc:2747] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:41.773962 5802 raft_consensus.cc:2747] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:41.774127 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8122b26848141a39d2462dd3491f9cd" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:41.781270 5807 raft_consensus.cc:513] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.782328 4979 raft_consensus.cc:2391] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:41.792124 5353 ts_tablet_manager.cc:1428] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.126s user 0.006s sys 0.000s
W20250626 19:57:41.785153 4756 leader_election.cc:343] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:41.793388 4756 leader_election.cc:304] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:41.794157 5772 raft_consensus.cc:2747] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:41.795372 5807 leader_election.cc:290] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.796553 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8122b26848141a39d2462dd3491f9cd" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:41.797111 5353 tablet_bootstrap.cc:492] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:41.800556 5410 tablet_bootstrap.cc:654] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:41.798918 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8122b26848141a39d2462dd3491f9cd" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:41.809468 4819 raft_consensus.cc:3058] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:41.810541 5782 ts_tablet_manager.cc:1918] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:57:41.815696 5366 tablet_bootstrap.cc:654] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:41.816030 4819 raft_consensus.cc:2466] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:41.817579 4820 raft_consensus.cc:2391] T e8122b26848141a39d2462dd3491f9cd P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:41.807228 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8122b26848141a39d2462dd3491f9cd" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:41.818462 4670 raft_consensus.cc:2391] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:41.823191 5782 log.cc:1199] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/6b66cda4f8ed410184b434d5083deac4
I20250626 19:57:41.826749 5623 raft_consensus.cc:491] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.827396 5623 raft_consensus.cc:513] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.830251 5433 raft_consensus.cc:2270] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:41.830281 5623 leader_election.cc:290] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:41.831110 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e65b16141c3d47c5b66024da59b9bbe9" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:41.834224 5772 consensus_queue.cc:1035] T 211f6137d9b84afa9aee6432440f7eb8 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:57:41.834815 4611 leader_election.cc:343] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:41.842875 5782 ts_tablet_manager.cc:1939] T 6b66cda4f8ed410184b434d5083deac4 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:41.853154 4609 leader_election.cc:304] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:41.854282 5433 ts_tablet_manager.cc:1905] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:41.855484 5623 raft_consensus.cc:2802] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:41.856086 5623 raft_consensus.cc:695] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:41.857717 5623 consensus_queue.cc:237] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.860219 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 6b66cda4f8ed410184b434d5083deac4 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]) successfully deleted
I20250626 19:57:41.861482 4609 leader_election.cc:304] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:41.866891 5689 raft_consensus.cc:2802] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:41.867605 5689 raft_consensus.cc:695] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:41.868633 5689 consensus_queue.cc:237] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.871939 4398 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:41.877127 5353 tablet_bootstrap.cc:654] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:41.891460 5410 tablet_bootstrap.cc:492] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:41.898730 5506 raft_consensus.cc:491] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:41.899210 5506 raft_consensus.cc:513] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:41.901122 5506 leader_election.cc:290] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:41.903108 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6ec952321924098a142f1438a283aab" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:41.907380 4398 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:41.910172 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6ec952321924098a142f1438a283aab" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:41.911742 4756 leader_election.cc:343] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:41.919374 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e65b16141c3d47c5b66024da59b9bbe9" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:41.921068 5366 tablet_bootstrap.cc:492] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:41.921460 5366 ts_tablet_manager.cc:1397] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.180s user 0.007s sys 0.004s
I20250626 19:57:41.920222 4817 raft_consensus.cc:2466] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:41.923076 4912 leader_election.cc:304] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:41.924373 5807 raft_consensus.cc:2747] T e8122b26848141a39d2462dd3491f9cd P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:57:41.926047 4398 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:41.930440 5366 raft_consensus.cc:357] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.938328 5366 raft_consensus.cc:383] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:41.938645 5366 raft_consensus.cc:738] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:41.944926 4398 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:41.947966 5366 consensus_queue.cc:260] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.960047 5366 ts_tablet_manager.cc:1428] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20250626 19:57:41.964248 5366 tablet_bootstrap.cc:492] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:41.969813 5366 tablet_bootstrap.cc:654] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:41.892010 5410 ts_tablet_manager.cc:1397] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.190s user 0.006s sys 0.006s
I20250626 19:57:41.993314 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:41.995935 5353 tablet_bootstrap.cc:492] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:41.996308 5353 ts_tablet_manager.cc:1397] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.199s user 0.013s sys 0.002s
I20250626 19:57:41.998406 5353 raft_consensus.cc:357] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:41.999250 5353 raft_consensus.cc:383] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:41.999573 5353 raft_consensus.cc:738] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.000172 5353 consensus_queue.cc:260] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.002681 5353 ts_tablet_manager.cc:1428] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250626 19:57:42.003751 5353 tablet_bootstrap.cc:492] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:42.008466 5807 raft_consensus.cc:491] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:42.008960 5807 raft_consensus.cc:513] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.011119 5807 leader_election.cc:290] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:42.012792 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1474bfa74a6346bf9e3ff49019f6c9b3" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:42.013597 4819 raft_consensus.cc:2466] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:42.015043 4911 leader_election.cc:304] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:42.016038 5807 raft_consensus.cc:2802] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:42.016450 5807 raft_consensus.cc:491] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:42.016844 5807 raft_consensus.cc:3058] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.021211 5433 ts_tablet_manager.cc:1918] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:57:42.021701 5433 log.cc:1199] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/6b66cda4f8ed410184b434d5083deac4
I20250626 19:57:42.022444 5433 ts_tablet_manager.cc:1939] T 6b66cda4f8ed410184b434d5083deac4 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:42.023862 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 6b66cda4f8ed410184b434d5083deac4 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]) successfully deleted
I20250626 19:57:42.030995 5410 raft_consensus.cc:357] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.031867 5410 raft_consensus.cc:383] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.032187 5410 raft_consensus.cc:738] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.057474 5366 tablet_bootstrap.cc:492] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:42.057484 5506 raft_consensus.cc:491] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:42.057956 5506 raft_consensus.cc:513] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:42.060561 4755 leader_election.cc:343] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:42.060691 5506 leader_election.cc:290] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:42.057955 5366 ts_tablet_manager.cc:1397] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.094s user 0.023s sys 0.007s
I20250626 19:57:42.061123 4755 leader_election.cc:304] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:42.063429 5366 raft_consensus.cc:357] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.064182 5366 raft_consensus.cc:383] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.064414 5366 raft_consensus.cc:738] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.064497 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb34ffcd4da04e32a1220b3f298ec22d" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:42.065798 5506 raft_consensus.cc:2747] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:42.066850 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb34ffcd4da04e32a1220b3f298ec22d" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:42.067019 5366 consensus_queue.cc:260] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:42.067910 4755 leader_election.cc:343] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:42.068382 4756 leader_election.cc:343] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:42.068711 4756 leader_election.cc:304] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:42.069865 5366 ts_tablet_manager.cc:1428] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250626 19:57:42.070039 5506 raft_consensus.cc:2747] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:42.070698 5366 tablet_bootstrap.cc:492] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
W20250626 19:57:42.074995 4609 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.4.46.188:40801, user_credentials={real_user=slave}} blocked reactor thread for 87996.5us
I20250626 19:57:42.081665 5353 tablet_bootstrap.cc:654] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.082486 4609 leader_election.cc:304] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:42.087903 5410 consensus_queue.cc:260] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.089850 5410 ts_tablet_manager.cc:1428] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.101s user 0.005s sys 0.000s
I20250626 19:57:42.090951 5802 raft_consensus.cc:2802] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:42.091425 5802 raft_consensus.cc:491] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:42.091814 5802 raft_consensus.cc:3058] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.078342 5807 raft_consensus.cc:513] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.093412 5807 leader_election.cc:290] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:42.098037 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1474bfa74a6346bf9e3ff49019f6c9b3" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:42.098519 4819 raft_consensus.cc:3058] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.099164 5366 tablet_bootstrap.cc:654] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.090675 5410 tablet_bootstrap.cc:492] T 465ed0fe56fb4960a58e92b9b3d594e6 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:42.103322 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1474bfa74a6346bf9e3ff49019f6c9b3" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:42.103955 4668 raft_consensus.cc:3058] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.106528 5410 tablet_bootstrap.cc:654] T 465ed0fe56fb4960a58e92b9b3d594e6 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.108903 5353 tablet_bootstrap.cc:492] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:42.109306 5353 ts_tablet_manager.cc:1397] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.106s user 0.005s sys 0.006s
I20250626 19:57:42.111554 5353 raft_consensus.cc:357] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.112253 5353 raft_consensus.cc:383] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.112605 5353 raft_consensus.cc:738] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.118180 5366 tablet_bootstrap.cc:492] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:42.118649 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1474bfa74a6346bf9e3ff49019f6c9b3" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:42.118824 5353 consensus_queue.cc:260] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.120164 5366 ts_tablet_manager.cc:1397] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.003s
I20250626 19:57:42.122988 5366 raft_consensus.cc:357] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.123628 5366 raft_consensus.cc:383] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.123879 5366 raft_consensus.cc:738] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.132126 5353 ts_tablet_manager.cc:1428] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20250626 19:57:42.132974 5353 tablet_bootstrap.cc:492] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:42.133046 5366 consensus_queue.cc:260] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.136611 5366 ts_tablet_manager.cc:1428] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250626 19:57:42.137429 5366 tablet_bootstrap.cc:492] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:42.153131 5366 tablet_bootstrap.cc:654] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.158573 4819 raft_consensus.cc:2466] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:42.159916 4911 leader_election.cc:304] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:42.160750 5794 raft_consensus.cc:2802] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:42.161196 5794 raft_consensus.cc:695] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:42.161906 5794 consensus_queue.cc:237] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.163409 5353 tablet_bootstrap.cc:654] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.163141 4668 raft_consensus.cc:2466] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:42.144295 5802 raft_consensus.cc:513] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.176738 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e65b16141c3d47c5b66024da59b9bbe9" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
W20250626 19:57:42.178826 4611 leader_election.cc:343] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:42.191496 5353 tablet_bootstrap.cc:492] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:42.191918 5353 ts_tablet_manager.cc:1397] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.059s user 0.012s sys 0.001s
I20250626 19:57:42.194110 5353 raft_consensus.cc:357] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.194725 5353 raft_consensus.cc:383] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.194984 5353 raft_consensus.cc:738] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.200577 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e65b16141c3d47c5b66024da59b9bbe9" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:42.201318 4819 raft_consensus.cc:3058] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.201637 5353 consensus_queue.cc:260] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.204605 5353 ts_tablet_manager.cc:1428] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s
I20250626 19:57:42.205476 5353 tablet_bootstrap.cc:492] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:42.214856 5802 leader_election.cc:290] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:42.217310 5366 tablet_bootstrap.cc:492] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:42.217712 5366 ts_tablet_manager.cc:1397] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.080s user 0.009s sys 0.000s
I20250626 19:57:42.220047 5366 raft_consensus.cc:357] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:42.220685 5366 raft_consensus.cc:383] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.220935 5366 raft_consensus.cc:738] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.221275 5410 tablet_bootstrap.cc:492] T 465ed0fe56fb4960a58e92b9b3d594e6 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:42.221709 5410 ts_tablet_manager.cc:1397] T 465ed0fe56fb4960a58e92b9b3d594e6 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.131s user 0.019s sys 0.021s
I20250626 19:57:42.223822 5410 raft_consensus.cc:357] T 465ed0fe56fb4960a58e92b9b3d594e6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.224414 5410 raft_consensus.cc:383] T 465ed0fe56fb4960a58e92b9b3d594e6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.224671 5410 raft_consensus.cc:738] T 465ed0fe56fb4960a58e92b9b3d594e6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.226888 5353 tablet_bootstrap.cc:654] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.228380 4957 tablet_service.cc:1468] Processing CreateTablet for tablet b852dd204f8a42cca629901723fc9247 (DEFAULT_TABLE table=default.Table_151fcc3e95fc46718ce92d0cfc4dcbaa [id=b4a09766d5a44355abccb84ca3c20920]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.228358 5410 consensus_queue.cc:260] T 465ed0fe56fb4960a58e92b9b3d594e6 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.230999 5410 ts_tablet_manager.cc:1428] T 465ed0fe56fb4960a58e92b9b3d594e6 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20250626 19:57:42.233688 5410 tablet_bootstrap.cc:492] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:42.234396 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b852dd204f8a42cca629901723fc9247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.235541 5366 consensus_queue.cc:260] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:42.245316 5366 ts_tablet_manager.cc:1428] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250626 19:57:42.247301 4648 tablet_service.cc:1468] Processing CreateTablet for tablet b852dd204f8a42cca629901723fc9247 (DEFAULT_TABLE table=default.Table_151fcc3e95fc46718ce92d0cfc4dcbaa [id=b4a09766d5a44355abccb84ca3c20920]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.248409 4648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b852dd204f8a42cca629901723fc9247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.248988 5807 raft_consensus.cc:491] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:42.249459 5807 raft_consensus.cc:513] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.251282 5807 leader_election.cc:290] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:42.251941 4952 tablet_service.cc:1468] Processing CreateTablet for tablet cf5c4cd6c2f64877a9e348dedb9820a3 (DEFAULT_TABLE table=default.Table_dcae42312225483a9c5104bf2551f652 [id=963772c3a795497c84226509c60b6f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.252947 4952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf5c4cd6c2f64877a9e348dedb9820a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.255085 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c113e03594b428a979435618fe373c6" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:42.255429 5366 tablet_bootstrap.cc:492] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:42.257871 5353 tablet_bootstrap.cc:492] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:42.258250 5353 ts_tablet_manager.cc:1397] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.002s
I20250626 19:57:42.258656 4797 tablet_service.cc:1468] Processing CreateTablet for tablet b852dd204f8a42cca629901723fc9247 (DEFAULT_TABLE table=default.Table_151fcc3e95fc46718ce92d0cfc4dcbaa [id=b4a09766d5a44355abccb84ca3c20920]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.259586 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c113e03594b428a979435618fe373c6" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:42.259910 4797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b852dd204f8a42cca629901723fc9247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.260282 4817 raft_consensus.cc:2391] T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:42.260542 5353 raft_consensus.cc:357] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.261159 5353 raft_consensus.cc:383] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.261273 4911 leader_election.cc:304] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:42.261400 5353 raft_consensus.cc:738] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.261952 5807 raft_consensus.cc:3058] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.267880 5410 tablet_bootstrap.cc:654] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.269399 5807 raft_consensus.cc:2747] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:42.270314 5353 consensus_queue.cc:260] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.271757 4656 tablet_service.cc:1468] Processing CreateTablet for tablet cf5c4cd6c2f64877a9e348dedb9820a3 (DEFAULT_TABLE table=default.Table_dcae42312225483a9c5104bf2551f652 [id=963772c3a795497c84226509c60b6f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.272714 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf5c4cd6c2f64877a9e348dedb9820a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.272794 5353 ts_tablet_manager.cc:1428] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.014s user 0.001s sys 0.005s
I20250626 19:57:42.272668 4799 tablet_service.cc:1468] Processing CreateTablet for tablet cf5c4cd6c2f64877a9e348dedb9820a3 (DEFAULT_TABLE table=default.Table_dcae42312225483a9c5104bf2551f652 [id=963772c3a795497c84226509c60b6f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.273605 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf5c4cd6c2f64877a9e348dedb9820a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.273591 5353 tablet_bootstrap.cc:492] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:42.287698 5807 raft_consensus.cc:491] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:42.288157 5807 raft_consensus.cc:513] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.290805 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ba78c1b40ef4925b4108e1455bcfc44" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:42.291808 4817 raft_consensus.cc:2391] T 1ba78c1b40ef4925b4108e1455bcfc44 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:42.294458 5353 tablet_bootstrap.cc:654] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.294184 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ba78c1b40ef4925b4108e1455bcfc44" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:42.295876 4912 leader_election.cc:304] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:42.297577 4795 tablet_service.cc:1468] Processing CreateTablet for tablet 075dbe46bab14d40a8008ce5c24bcf38 (DEFAULT_TABLE table=default.Table_d6eed56e9d514833bfe29ff3aba9718a [id=07c980b0511a43c0aa406597788cd05b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.298050 4649 tablet_service.cc:1468] Processing CreateTablet for tablet 075dbe46bab14d40a8008ce5c24bcf38 (DEFAULT_TABLE table=default.Table_d6eed56e9d514833bfe29ff3aba9718a [id=07c980b0511a43c0aa406597788cd05b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.298543 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075dbe46bab14d40a8008ce5c24bcf38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.299362 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075dbe46bab14d40a8008ce5c24bcf38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.302850 5794 raft_consensus.cc:3058] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.307123 5773 raft_consensus.cc:491] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:42.307574 5773 raft_consensus.cc:513] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.309593 5623 consensus_queue.cc:1035] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:42.310256 5773 leader_election.cc:290] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:42.311475 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69bd106066b6492bb7ea03d15a18df00" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:42.312009 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69bd106066b6492bb7ea03d15a18df00" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:42.312624 4755 leader_election.cc:343] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:42.314647 4756 leader_election.cc:343] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:42.315032 4756 leader_election.cc:304] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:42.316567 5773 raft_consensus.cc:2747] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:42.318411 5807 leader_election.cc:290] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:42.324081 5794 raft_consensus.cc:2747] T 1ba78c1b40ef4925b4108e1455bcfc44 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:42.333160 4951 tablet_service.cc:1468] Processing CreateTablet for tablet 075dbe46bab14d40a8008ce5c24bcf38 (DEFAULT_TABLE table=default.Table_d6eed56e9d514833bfe29ff3aba9718a [id=07c980b0511a43c0aa406597788cd05b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.334482 4951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075dbe46bab14d40a8008ce5c24bcf38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.335475 5410 tablet_bootstrap.cc:492] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:42.335870 5410 ts_tablet_manager.cc:1397] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.102s user 0.016s sys 0.008s
I20250626 19:57:42.341925 5410 raft_consensus.cc:357] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.342571 5410 raft_consensus.cc:383] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.342831 5410 raft_consensus.cc:738] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.348408 5410 consensus_queue.cc:260] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.354022 5623 consensus_queue.cc:1035] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:42.358886 5410 ts_tablet_manager.cc:1428] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20250626 19:57:42.362104 5410 tablet_bootstrap.cc:492] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:42.368671 4819 raft_consensus.cc:2466] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:42.369727 4609 leader_election.cc:304] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:42.370735 5623 raft_consensus.cc:2802] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:42.371218 5623 raft_consensus.cc:695] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:42.372000 5623 consensus_queue.cc:237] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.377027 5353 tablet_bootstrap.cc:492] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:42.380529 5353 ts_tablet_manager.cc:1397] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.107s user 0.008s sys 0.003s
I20250626 19:57:42.383025 5353 raft_consensus.cc:357] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.383412 4942 tablet_service.cc:1468] Processing CreateTablet for tablet 27acc0eafbb04375ba1c6a59e958e6f6 (DEFAULT_TABLE table=default.Table_c8720865c8c742bc941a7d2665c618f5 [id=30ad5642ec2d422aad5b97698297fbe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.384001 5410 tablet_bootstrap.cc:654] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.383661 5353 raft_consensus.cc:383] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.384315 5353 raft_consensus.cc:738] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.384452 4942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27acc0eafbb04375ba1c6a59e958e6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.384891 5353 consensus_queue.cc:260] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.389806 5353 ts_tablet_manager.cc:1428] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20250626 19:57:42.399819 5353 tablet_bootstrap.cc:492] T a3636b78f4a64f9eacb4e5979768929b P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:42.407797 4653 tablet_service.cc:1468] Processing CreateTablet for tablet 27acc0eafbb04375ba1c6a59e958e6f6 (DEFAULT_TABLE table=default.Table_c8720865c8c742bc941a7d2665c618f5 [id=30ad5642ec2d422aad5b97698297fbe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.411607 4793 tablet_service.cc:1468] Processing CreateTablet for tablet 27acc0eafbb04375ba1c6a59e958e6f6 (DEFAULT_TABLE table=default.Table_c8720865c8c742bc941a7d2665c618f5 [id=30ad5642ec2d422aad5b97698297fbe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.412668 4793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27acc0eafbb04375ba1c6a59e958e6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.421496 5353 tablet_bootstrap.cc:654] T a3636b78f4a64f9eacb4e5979768929b P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.439280 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:42.442879 5366 tablet_bootstrap.cc:654] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.445470 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27acc0eafbb04375ba1c6a59e958e6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.447300 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:42.449836 4950 tablet_service.cc:1468] Processing CreateTablet for tablet 1feb6785d0c7459fbc6898b0c9d3e38c (DEFAULT_TABLE table=default.Table_987278d268e04477828ad87c6d9c8c01 [id=da221881c13940a4b0dfeb88aa81c932]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.450801 4950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1feb6785d0c7459fbc6898b0c9d3e38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.463268 5794 raft_consensus.cc:491] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:42.463907 5794 raft_consensus.cc:513] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.465961 5794 leader_election.cc:290] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:42.472205 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "465ed0fe56fb4960a58e92b9b3d594e6" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:42.472820 4819 raft_consensus.cc:2466] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:42.474572 4911 leader_election.cc:304] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:42.475418 5794 raft_consensus.cc:2802] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:42.475817 5794 raft_consensus.cc:491] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:42.476310 5794 raft_consensus.cc:3058] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.487362 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "465ed0fe56fb4960a58e92b9b3d594e6" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:42.487927 4675 raft_consensus.cc:2466] T 465ed0fe56fb4960a58e92b9b3d594e6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:42.488853 5410 tablet_bootstrap.cc:492] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:42.489235 5410 ts_tablet_manager.cc:1397] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.127s user 0.017s sys 0.013s
I20250626 19:57:42.492812 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 1feb6785d0c7459fbc6898b0c9d3e38c (DEFAULT_TABLE table=default.Table_987278d268e04477828ad87c6d9c8c01 [id=da221881c13940a4b0dfeb88aa81c932]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.492945 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 159a6bf746e14479b3f3584640b63098 (DEFAULT_TABLE table=default.Table_f6018bf185744bb9981e34b1c75cc712 [id=69636c9141f04e09a115427ab6b91c3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.493723 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1feb6785d0c7459fbc6898b0c9d3e38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.494240 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 159a6bf746e14479b3f3584640b63098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.493948 5410 raft_consensus.cc:357] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.494536 5410 raft_consensus.cc:383] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.497488 4651 tablet_service.cc:1468] Processing CreateTablet for tablet 1feb6785d0c7459fbc6898b0c9d3e38c (DEFAULT_TABLE table=default.Table_987278d268e04477828ad87c6d9c8c01 [id=da221881c13940a4b0dfeb88aa81c932]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.501811 4651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1feb6785d0c7459fbc6898b0c9d3e38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.498395 4652 tablet_service.cc:1468] Processing CreateTablet for tablet 159a6bf746e14479b3f3584640b63098 (DEFAULT_TABLE table=default.Table_f6018bf185744bb9981e34b1c75cc712 [id=69636c9141f04e09a115427ab6b91c3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.502547 5410 raft_consensus.cc:738] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.503217 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 159a6bf746e14479b3f3584640b63098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.503309 5410 consensus_queue.cc:260] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.506614 5410 ts_tablet_manager.cc:1428] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.017s user 0.001s sys 0.005s
I20250626 19:57:42.507429 5410 tablet_bootstrap.cc:492] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:42.513128 5794 raft_consensus.cc:513] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.517095 5794 leader_election.cc:290] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:42.523056 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "465ed0fe56fb4960a58e92b9b3d594e6" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:42.523607 4668 raft_consensus.cc:3058] T 465ed0fe56fb4960a58e92b9b3d594e6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.525091 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "465ed0fe56fb4960a58e92b9b3d594e6" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:42.525642 4819 raft_consensus.cc:3058] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.532830 4797 tablet_service.cc:1468] Processing CreateTablet for tablet de0724ba71824a5daaf6f3708b4431b7 (DEFAULT_TABLE table=default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.535030 5353 tablet_bootstrap.cc:492] T a3636b78f4a64f9eacb4e5979768929b P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:42.535458 5353 ts_tablet_manager.cc:1397] T a3636b78f4a64f9eacb4e5979768929b P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.136s user 0.008s sys 0.010s
I20250626 19:57:42.537676 5353 raft_consensus.cc:357] T a3636b78f4a64f9eacb4e5979768929b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.538358 5353 raft_consensus.cc:383] T a3636b78f4a64f9eacb4e5979768929b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.540949 4650 tablet_service.cc:1468] Processing CreateTablet for tablet de0724ba71824a5daaf6f3708b4431b7 (DEFAULT_TABLE table=default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]), partition=RANGE (key) PARTITION UNBOUNDED
W20250626 19:57:42.543154 4718 debug-util.cc:398] Leaking SignalData structure 0x7b08000f3b60 after lost signal to thread 4581
I20250626 19:57:42.544078 4650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de0724ba71824a5daaf6f3708b4431b7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:42.544636 4718 debug-util.cc:398] Leaking SignalData structure 0x7b080003d900 after lost signal to thread 4721
I20250626 19:57:42.533245 4819 raft_consensus.cc:2466] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:42.550981 4797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de0724ba71824a5daaf6f3708b4431b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.557957 4911 leader_election.cc:304] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:42.559170 5410 tablet_bootstrap.cc:654] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.538661 5353 raft_consensus.cc:738] T a3636b78f4a64f9eacb4e5979768929b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.553211 5366 tablet_bootstrap.cc:492] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:42.561173 5366 ts_tablet_manager.cc:1397] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.306s user 0.010s sys 0.000s
I20250626 19:57:42.560286 5353 consensus_queue.cc:260] T a3636b78f4a64f9eacb4e5979768929b P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.564512 5807 raft_consensus.cc:2802] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:42.565126 5807 raft_consensus.cc:695] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:42.566094 5807 consensus_queue.cc:237] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.569264 4954 tablet_service.cc:1468] Processing CreateTablet for tablet 8ea7179912284fef8d2165cdfaf4a21a (DEFAULT_TABLE table=default.Table_16e44e41e06c4d5f8e08d73dc5fa185e [id=4ad299909ad54ed2a884c8643c377d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.570163 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ea7179912284fef8d2165cdfaf4a21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.571858 4942 tablet_service.cc:1468] Processing CreateTablet for tablet de0724ba71824a5daaf6f3708b4431b7 (DEFAULT_TABLE table=default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.572690 5366 raft_consensus.cc:357] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.579190 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:42.579876 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:42.580953 4390 catalog_manager.cc:5582] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:42.582931 5366 raft_consensus.cc:383] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.583257 5366 raft_consensus.cc:738] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.583338 5802 consensus_queue.cc:1035] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:42.584326 5366 consensus_queue.cc:260] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.593665 5366 ts_tablet_manager.cc:1428] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20250626 19:57:42.607033 4942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de0724ba71824a5daaf6f3708b4431b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.607877 5794 raft_consensus.cc:491] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:42.608702 5794 raft_consensus.cc:513] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.613303 5794 leader_election.cc:290] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:42.560890 4957 tablet_service.cc:1468] Processing CreateTablet for tablet 159a6bf746e14479b3f3584640b63098 (DEFAULT_TABLE table=default.Table_f6018bf185744bb9981e34b1c75cc712 [id=69636c9141f04e09a115427ab6b91c3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.615595 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 159a6bf746e14479b3f3584640b63098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.601902 5353 ts_tablet_manager.cc:1428] T a3636b78f4a64f9eacb4e5979768929b P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.066s user 0.007s sys 0.000s
I20250626 19:57:42.616484 5353 tablet_bootstrap.cc:492] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:42.618342 4952 tablet_service.cc:1468] Processing CreateTablet for tablet 4a6058961bc7450ab489b26158e7dd17 (DEFAULT_TABLE table=default.Table_889f18064aab4774b17bfa478a0a7e33 [id=0df063bc889246c3ba841b1b2f8e6104]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.605847 4795 tablet_service.cc:1468] Processing CreateTablet for tablet 4a6058961bc7450ab489b26158e7dd17 (DEFAULT_TABLE table=default.Table_889f18064aab4774b17bfa478a0a7e33 [id=0df063bc889246c3ba841b1b2f8e6104]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.635301 4950 tablet_service.cc:1468] Processing CreateTablet for tablet f70c620bf4474e59aa1f86761aca0daa (DEFAULT_TABLE table=default.Table_56d52ecce53b4f1e8afea2f6d1695a92 [id=780eb12091bc47b68832a3a33ea69ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.635774 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6058961bc7450ab489b26158e7dd17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.636623 4950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f70c620bf4474e59aa1f86761aca0daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.636951 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:42.646171 4952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6058961bc7450ab489b26158e7dd17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.653865 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 8ea7179912284fef8d2165cdfaf4a21a (DEFAULT_TABLE table=default.Table_16e44e41e06c4d5f8e08d73dc5fa185e [id=4ad299909ad54ed2a884c8643c377d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.654946 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ea7179912284fef8d2165cdfaf4a21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.652426 4657 tablet_service.cc:1468] Processing CreateTablet for tablet 4a6058961bc7450ab489b26158e7dd17 (DEFAULT_TABLE table=default.Table_889f18064aab4774b17bfa478a0a7e33 [id=0df063bc889246c3ba841b1b2f8e6104]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.661018 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6058961bc7450ab489b26158e7dd17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.667076 4654 tablet_service.cc:1468] Processing CreateTablet for tablet f70c620bf4474e59aa1f86761aca0daa (DEFAULT_TABLE table=default.Table_56d52ecce53b4f1e8afea2f6d1695a92 [id=780eb12091bc47b68832a3a33ea69ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.668119 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f70c620bf4474e59aa1f86761aca0daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.667969 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e65b16141c3d47c5b66024da59b9bbe9" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:42.668638 4820 raft_consensus.cc:2391] T e65b16141c3d47c5b66024da59b9bbe9 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:42.669425 5506 raft_consensus.cc:491] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:42.670019 4801 tablet_service.cc:1468] Processing CreateTablet for tablet f70c620bf4474e59aa1f86761aca0daa (DEFAULT_TABLE table=default.Table_56d52ecce53b4f1e8afea2f6d1695a92 [id=780eb12091bc47b68832a3a33ea69ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.669836 5506 raft_consensus.cc:513] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.672158 5506 leader_election.cc:290] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:42.672725 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "781f8ee8c8ad49c6b2391fcdf20bdfdb" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:42.673079 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f70c620bf4474e59aa1f86761aca0daa. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:42.674018 4756 leader_election.cc:343] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:42.675657 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "781f8ee8c8ad49c6b2391fcdf20bdfdb" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:42.677039 4755 leader_election.cc:343] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:42.677404 4755 leader_election.cc:304] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:42.678429 5506 raft_consensus.cc:2747] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:42.678314 4793 tablet_service.cc:1468] Processing CreateTablet for tablet 8ea7179912284fef8d2165cdfaf4a21a (DEFAULT_TABLE table=default.Table_16e44e41e06c4d5f8e08d73dc5fa185e [id=4ad299909ad54ed2a884c8643c377d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:42.689450 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e65b16141c3d47c5b66024da59b9bbe9" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:42.693188 5366 tablet_bootstrap.cc:492] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:42.697726 4668 raft_consensus.cc:2466] T 465ed0fe56fb4960a58e92b9b3d594e6 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:42.709647 4912 leader_election.cc:304] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:42.710314 5851 raft_consensus.cc:3058] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.713004 5807 raft_consensus.cc:491] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:42.713480 5807 raft_consensus.cc:513] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.627905 5353 tablet_bootstrap.cc:654] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.715334 5807 leader_election.cc:290] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:42.718266 5807 consensus_queue.cc:1035] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:42.718883 5410 tablet_bootstrap.cc:492] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:42.719524 5410 ts_tablet_manager.cc:1397] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.212s user 0.008s sys 0.005s
I20250626 19:57:42.719902 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5706841a14bd4ece965098b88ccb7ec8" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:42.721777 5410 raft_consensus.cc:357] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.722354 5410 raft_consensus.cc:383] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.722616 5410 raft_consensus.cc:738] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.723160 5410 consensus_queue.cc:260] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.739825 5410 ts_tablet_manager.cc:1428] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20250626 19:57:42.740684 5410 tablet_bootstrap.cc:492] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:42.743896 5851 raft_consensus.cc:2747] T e65b16141c3d47c5b66024da59b9bbe9 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:42.745966 5772 raft_consensus.cc:491] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:42.747149 5772 raft_consensus.cc:513] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.745932 5506 raft_consensus.cc:491] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:42.749197 5506 raft_consensus.cc:513] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:42.751042 5506 leader_election.cc:290] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:42.752295 4978 raft_consensus.cc:3058] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.758498 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3143181fae16411888d74f9152f3d5cc" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:42.759840 4755 leader_election.cc:343] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:42.760114 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3143181fae16411888d74f9152f3d5cc" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:42.760541 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "609d36d46bb84017b2a3a7ad48bfccfc" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:42.760902 5802 raft_consensus.cc:491] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 19:57:42.761067 4756 leader_election.cc:343] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:42.761418 4756 leader_election.cc:304] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:42.761313 5802 raft_consensus.cc:513] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.761868 4793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ea7179912284fef8d2165cdfaf4a21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:42.761737 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5706841a14bd4ece965098b88ccb7ec8" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:42.763015 5802 leader_election.cc:290] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:42.762989 5506 raft_consensus.cc:2747] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:42.767071 4756 leader_election.cc:343] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:42.767320 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3636b78f4a64f9eacb4e5979768929b" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:42.767796 4911 leader_election.cc:304] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:42.767830 4976 raft_consensus.cc:2466] T a3636b78f4a64f9eacb4e5979768929b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:42.769083 5798 consensus_queue.cc:1035] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:42.770999 4611 leader_election.cc:304] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:42.771780 5802 raft_consensus.cc:2802] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:42.772089 5802 raft_consensus.cc:491] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:42.772400 5802 raft_consensus.cc:3058] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.749191 5772 leader_election.cc:290] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:42.779798 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3636b78f4a64f9eacb4e5979768929b" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:42.780407 4815 raft_consensus.cc:2466] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:42.796711 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "609d36d46bb84017b2a3a7ad48bfccfc" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:42.796031 5807 raft_consensus.cc:2747] T 5706841a14bd4ece965098b88ccb7ec8 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:42.797923 4755 leader_election.cc:343] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:42.798314 4755 leader_election.cc:304] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:42.799005 5689 consensus_queue.cc:1035] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:42.802037 5506 raft_consensus.cc:2747] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:42.806152 5410 tablet_bootstrap.cc:654] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.812506 5353 tablet_bootstrap.cc:492] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:42.812964 5353 ts_tablet_manager.cc:1397] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.197s user 0.004s sys 0.004s
I20250626 19:57:42.810927 5802 raft_consensus.cc:513] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.823335 5623 consensus_queue.cc:1035] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:42.828946 5366 tablet_bootstrap.cc:654] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.840282 5506 raft_consensus.cc:491] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:42.840713 5506 raft_consensus.cc:513] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.841629 5802 leader_election.cc:290] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:42.843737 5506 leader_election.cc:290] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:42.847154 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3636b78f4a64f9eacb4e5979768929b" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:42.847496 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3636b78f4a64f9eacb4e5979768929b" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:42.847698 4820 raft_consensus.cc:3058] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.848028 4976 raft_consensus.cc:3058] T a3636b78f4a64f9eacb4e5979768929b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:42.849645 5353 raft_consensus.cc:357] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.850271 5353 raft_consensus.cc:383] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.850529 5353 raft_consensus.cc:738] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.851116 5353 consensus_queue.cc:260] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.854108 4820 raft_consensus.cc:2466] T a3636b78f4a64f9eacb4e5979768929b P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:42.855886 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bf4c4596b4441839c82b5fe5c5eea37" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:42.857620 4756 leader_election.cc:343] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:42.857925 4609 leader_election.cc:304] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:42.859402 5689 raft_consensus.cc:2802] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:42.859942 5689 raft_consensus.cc:695] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:42.860773 5689 consensus_queue.cc:237] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.871579 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:42.877553 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bf4c4596b4441839c82b5fe5c5eea37" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:42.870851 5353 ts_tablet_manager.cc:1428] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.058s user 0.008s sys 0.000s
W20250626 19:57:42.879205 4755 leader_election.cc:343] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:42.879586 4755 leader_election.cc:304] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:42.880472 5506 raft_consensus.cc:2747] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:42.889174 5353 tablet_bootstrap.cc:492] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:42.906639 5353 tablet_bootstrap.cc:654] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.915261 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:42.915781 4976 raft_consensus.cc:2466] T a3636b78f4a64f9eacb4e5979768929b P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:42.921940 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:42.927392 5353 tablet_bootstrap.cc:492] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:42.927788 5353 ts_tablet_manager.cc:1397] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.000s
I20250626 19:57:42.929991 5353 raft_consensus.cc:357] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:42.930692 5353 raft_consensus.cc:383] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.931039 5353 raft_consensus.cc:738] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.939391 5353 consensus_queue.cc:260] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:42.945693 5623 consensus_queue.cc:1035] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:42.948347 5353 ts_tablet_manager.cc:1428] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20250626 19:57:42.949582 5353 tablet_bootstrap.cc:492] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:42.952144 5366 tablet_bootstrap.cc:492] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:42.952530 5366 ts_tablet_manager.cc:1397] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.260s user 0.013s sys 0.007s
I20250626 19:57:42.954836 5366 raft_consensus.cc:357] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.955412 5366 raft_consensus.cc:383] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.955646 5366 raft_consensus.cc:738] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.957125 5366 consensus_queue.cc:260] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:42.961164 5353 tablet_bootstrap.cc:654] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:42.930430 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:42.966325 5410 tablet_bootstrap.cc:492] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:42.966802 5410 ts_tablet_manager.cc:1397] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.226s user 0.001s sys 0.007s
I20250626 19:57:42.969074 5410 raft_consensus.cc:357] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:42.969692 5410 raft_consensus.cc:383] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:42.969940 5410 raft_consensus.cc:738] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:42.971951 5623 consensus_queue.cc:1035] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:42.984201 5366 ts_tablet_manager.cc:1428] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.031s user 0.005s sys 0.001s
I20250626 19:57:42.999821 5410 consensus_queue.cc:260] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:43.002036 5410 ts_tablet_manager.cc:1428] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.035s user 0.004s sys 0.001s
I20250626 19:57:43.013716 5807 raft_consensus.cc:491] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:43.014235 5807 raft_consensus.cc:513] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.016359 5807 leader_election.cc:290] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:43.017429 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb34ffcd4da04e32a1220b3f298ec22d" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:43.017954 4819 raft_consensus.cc:2466] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:43.019135 4911 leader_election.cc:304] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:43.020366 5366 tablet_bootstrap.cc:492] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:43.020809 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb34ffcd4da04e32a1220b3f298ec22d" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:43.021384 4675 raft_consensus.cc:2466] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:43.023833 5807 raft_consensus.cc:2802] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:43.024302 5807 raft_consensus.cc:491] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:43.024535 5410 tablet_bootstrap.cc:492] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:43.024701 5807 raft_consensus.cc:3058] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:43.033337 5851 consensus_queue.cc:1035] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:43.044229 5410 tablet_bootstrap.cc:654] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:43.049317 5794 consensus_queue.cc:1035] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:43.050544 5366 tablet_bootstrap.cc:654] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:43.063961 5623 consensus_queue.cc:1035] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:43.084059 5807 raft_consensus.cc:513] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.087936 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb34ffcd4da04e32a1220b3f298ec22d" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:43.088032 5807 leader_election.cc:290] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:43.088485 4820 raft_consensus.cc:3058] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:43.101595 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb34ffcd4da04e32a1220b3f298ec22d" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:43.102454 4668 raft_consensus.cc:3058] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:43.105816 4820 raft_consensus.cc:2466] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:43.106911 4911 leader_election.cc:304] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:43.109067 5890 raft_consensus.cc:2802] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:43.110997 5410 tablet_bootstrap.cc:492] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:43.113289 5890 raft_consensus.cc:695] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:43.113862 5410 ts_tablet_manager.cc:1397] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.111s user 0.011s sys 0.000s
I20250626 19:57:43.115276 5890 consensus_queue.cc:237] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.128957 5366 tablet_bootstrap.cc:492] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:43.129586 5366 ts_tablet_manager.cc:1397] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.109s user 0.004s sys 0.008s
I20250626 19:57:43.132094 5366 raft_consensus.cc:357] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.132828 5366 raft_consensus.cc:383] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:43.133152 5366 raft_consensus.cc:738] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:43.133661 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:43.135604 5410 raft_consensus.cc:357] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.136255 5410 raft_consensus.cc:383] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:43.136502 5410 raft_consensus.cc:738] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:43.137099 5410 consensus_queue.cc:260] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.116570 4668 raft_consensus.cc:2466] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:43.163271 5366 consensus_queue.cc:260] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.165694 5366 ts_tablet_manager.cc:1428] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.036s user 0.004s sys 0.000s
I20250626 19:57:43.166515 5366 tablet_bootstrap.cc:492] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:43.170918 5353 tablet_bootstrap.cc:492] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:43.171502 5353 ts_tablet_manager.cc:1397] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.222s user 0.011s sys 0.000s
I20250626 19:57:43.174121 5353 raft_consensus.cc:357] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.138587 5410 ts_tablet_manager.cc:1428] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250626 19:57:43.182192 5353 raft_consensus.cc:383] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:43.182603 5353 raft_consensus.cc:738] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:43.185927 5353 consensus_queue.cc:260] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.192108 5410 tablet_bootstrap.cc:492] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:43.193872 5353 ts_tablet_manager.cc:1428] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.022s user 0.004s sys 0.003s
I20250626 19:57:43.194813 5353 tablet_bootstrap.cc:492] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:43.206168 5366 tablet_bootstrap.cc:654] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:43.209451 5410 tablet_bootstrap.cc:654] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:43.209594 5353 tablet_bootstrap.cc:654] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:43.221626 5353 tablet_bootstrap.cc:492] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:43.222265 5353 ts_tablet_manager.cc:1397] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.001s
I20250626 19:57:43.226202 5353 raft_consensus.cc:357] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:43.230181 5798 raft_consensus.cc:491] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:43.234879 5353 raft_consensus.cc:383] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:43.234978 5266 rpc_server.cc:307] RPC server started. Bound to: 127.4.46.190:33823
I20250626 19:57:43.235167 5353 raft_consensus.cc:738] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:43.235090 5798 raft_consensus.cc:513] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:43.235870 5353 consensus_queue.cc:260] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:43.237344 5798 leader_election.cc:290] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:43.238040 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6ec952321924098a142f1438a283aab" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:43.238662 4668 raft_consensus.cc:2466] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:43.239723 4912 leader_election.cc:304] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:43.234586 5903 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.46.190:33823 every 8 connection(s)
I20250626 19:57:43.240346 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6ec952321924098a142f1438a283aab" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:43.240480 5266 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/master-0/data/info.pb
I20250626 19:57:43.240862 4819 raft_consensus.cc:2466] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:43.241828 4282 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu as pid 5266
I20250626 19:57:43.240795 5890 raft_consensus.cc:2802] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:43.246201 5890 raft_consensus.cc:491] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:43.246606 5890 raft_consensus.cc:3058] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:43.268463 5802 consensus_queue.cc:1035] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:43.283695 5890 raft_consensus.cc:513] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:43.286100 5890 leader_election.cc:290] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:43.287020 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6ec952321924098a142f1438a283aab" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:43.287266 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6ec952321924098a142f1438a283aab" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:43.287539 4819 raft_consensus.cc:3058] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:43.287825 4668 raft_consensus.cc:3058] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:43.289315 5366 tablet_bootstrap.cc:492] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:43.289752 5366 ts_tablet_manager.cc:1397] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.123s user 0.009s sys 0.001s
I20250626 19:57:43.292418 5366 raft_consensus.cc:357] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.293284 5366 raft_consensus.cc:383] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:43.293751 5907 sys_catalog.cc:263] Verifying existing consensus state
I20250626 19:57:43.294945 5410 tablet_bootstrap.cc:492] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:43.295374 5410 ts_tablet_manager.cc:1397] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.103s user 0.010s sys 0.000s
I20250626 19:57:43.297078 4668 raft_consensus.cc:2466] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:43.297863 5410 raft_consensus.cc:357] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.298853 5410 raft_consensus.cc:383] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:43.304517 5907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Bootstrap starting.
I20250626 19:57:43.298130 4912 leader_election.cc:304] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:43.306563 4952 tablet_service.cc:1468] Processing CreateTablet for tablet d668a84475064a3eaa08b0c719471c26 (DEFAULT_TABLE table=default.Table_f26100f290884036a30e2b11e0826db6 [id=dfcb0ed35c1447e892413939f43d39a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.336838 4952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d668a84475064a3eaa08b0c719471c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.314579 4950 tablet_service.cc:1468] Processing CreateTablet for tablet e953f25481a245dd8edd61cd2e6061d1 (DEFAULT_TABLE table=default.Table_7490b89a33c842b0853726a36f5cebe6 [id=82bfc17451224ad0a521e05e95d05d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.339452 4950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e953f25481a245dd8edd61cd2e6061d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.339743 5890 raft_consensus.cc:2802] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:43.340415 5890 raft_consensus.cc:695] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:43.341123 5890 consensus_queue.cc:237] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:43.317471 4957 tablet_service.cc:1468] Processing CreateTablet for tablet 0e6136b0d3104ae88c3f4ae88a41de84 (DEFAULT_TABLE table=default.Table_ab8780ef94764aadbd99279e11543fd2 [id=937847e0a37b4c4f89297045157373da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.343361 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e6136b0d3104ae88c3f4ae88a41de84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.357280 5410 raft_consensus.cc:738] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:43.307407 4656 tablet_service.cc:1468] Processing CreateTablet for tablet d668a84475064a3eaa08b0c719471c26 (DEFAULT_TABLE table=default.Table_f26100f290884036a30e2b11e0826db6 [id=dfcb0ed35c1447e892413939f43d39a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.358582 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d668a84475064a3eaa08b0c719471c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.357504 4951 tablet_service.cc:1468] Processing CreateTablet for tablet 7851e464fcd44ecb82404d6a991db2a2 (DEFAULT_TABLE table=default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.359589 4951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7851e464fcd44ecb82404d6a991db2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.359114 5410 consensus_queue.cc:260] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.362097 5410 ts_tablet_manager.cc:1428] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.066s user 0.005s sys 0.000s
I20250626 19:57:43.315317 4654 tablet_service.cc:1468] Processing CreateTablet for tablet e953f25481a245dd8edd61cd2e6061d1 (DEFAULT_TABLE table=default.Table_7490b89a33c842b0853726a36f5cebe6 [id=82bfc17451224ad0a521e05e95d05d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.362911 5410 tablet_bootstrap.cc:492] T 5bf4c4596b4441839c82b5fe5c5eea37 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:43.363438 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e953f25481a245dd8edd61cd2e6061d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.293558 4819 raft_consensus.cc:2466] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:43.318920 4657 tablet_service.cc:1468] Processing CreateTablet for tablet 0e6136b0d3104ae88c3f4ae88a41de84 (DEFAULT_TABLE table=default.Table_ab8780ef94764aadbd99279e11543fd2 [id=937847e0a37b4c4f89297045157373da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.311941 4793 tablet_service.cc:1468] Processing CreateTablet for tablet d668a84475064a3eaa08b0c719471c26 (DEFAULT_TABLE table=default.Table_f26100f290884036a30e2b11e0826db6 [id=dfcb0ed35c1447e892413939f43d39a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.315476 4797 tablet_service.cc:1468] Processing CreateTablet for tablet e953f25481a245dd8edd61cd2e6061d1 (DEFAULT_TABLE table=default.Table_7490b89a33c842b0853726a36f5cebe6 [id=82bfc17451224ad0a521e05e95d05d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.367398 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e6136b0d3104ae88c3f4ae88a41de84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.367633 5366 raft_consensus.cc:738] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:43.328354 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 54e68eba747e4a2a947e84e53a444d08 (DEFAULT_TABLE table=default.Table_8ef072e9d240402397d79db39729e650 [id=211696a0b141485ab4e4e2b1d190baa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.349510 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 8282e22bfb8b4d4994fa8b95ef154a68 (DEFAULT_TABLE table=default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.368296 5366 consensus_queue.cc:260] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.370880 5366 ts_tablet_manager.cc:1428] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.081s user 0.002s sys 0.004s
I20250626 19:57:43.371805 5366 tablet_bootstrap.cc:492] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:43.357715 4958 tablet_service.cc:1468] Processing CreateTablet for tablet 1b794133c1a346318e31bffb4ba6effb (DEFAULT_TABLE table=default.Table_d51d8fa1843f4e47a913158768b4689c [id=574f9c9a69d34e918b567ac099e081e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.352447 4794 tablet_service.cc:1468] Processing CreateTablet for tablet 7851e464fcd44ecb82404d6a991db2a2 (DEFAULT_TABLE table=default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.356747 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 1b794133c1a346318e31bffb4ba6effb (DEFAULT_TABLE table=default.Table_d51d8fa1843f4e47a913158768b4689c [id=574f9c9a69d34e918b567ac099e081e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.376515 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b794133c1a346318e31bffb4ba6effb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.367750 4797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e953f25481a245dd8edd61cd2e6061d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.317734 4795 tablet_service.cc:1468] Processing CreateTablet for tablet 0e6136b0d3104ae88c3f4ae88a41de84 (DEFAULT_TABLE table=default.Table_ab8780ef94764aadbd99279e11543fd2 [id=937847e0a37b4c4f89297045157373da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.379279 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54e68eba747e4a2a947e84e53a444d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.327057 4652 tablet_service.cc:1468] Processing CreateTablet for tablet 54e68eba747e4a2a947e84e53a444d08 (DEFAULT_TABLE table=default.Table_8ef072e9d240402397d79db39729e650 [id=211696a0b141485ab4e4e2b1d190baa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.383627 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54e68eba747e4a2a947e84e53a444d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.328811 4650 tablet_service.cc:1468] Processing CreateTablet for tablet 8282e22bfb8b4d4994fa8b95ef154a68 (DEFAULT_TABLE table=default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.387634 4650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8282e22bfb8b4d4994fa8b95ef154a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.391153 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8282e22bfb8b4d4994fa8b95ef154a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.391681 5506 raft_consensus.cc:491] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:43.392158 5506 raft_consensus.cc:513] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.393939 5506 leader_election.cc:290] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:43.395035 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b794133c1a346318e31bffb4ba6effb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.399065 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e6136b0d3104ae88c3f4ae88a41de84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.399143 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8be634cb36d40dca3f9338dab93ad2c" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:43.403146 4794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7851e464fcd44ecb82404d6a991db2a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:43.403191 4755 leader_election.cc:343] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:43.405472 4651 tablet_service.cc:1468] Processing CreateTablet for tablet 7851e464fcd44ecb82404d6a991db2a2 (DEFAULT_TABLE table=default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.405759 4649 tablet_service.cc:1468] Processing CreateTablet for tablet 1b794133c1a346318e31bffb4ba6effb (DEFAULT_TABLE table=default.Table_d51d8fa1843f4e47a913158768b4689c [id=574f9c9a69d34e918b567ac099e081e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.406419 4651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7851e464fcd44ecb82404d6a991db2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.407830 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b794133c1a346318e31bffb4ba6effb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.357714 4954 tablet_service.cc:1468] Processing CreateTablet for tablet 8282e22bfb8b4d4994fa8b95ef154a68 (DEFAULT_TABLE table=default.Table_654b4c156965473190b6f9ae638705af [id=ac9c273d435f4674b1299de6b451a6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.420010 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8282e22bfb8b4d4994fa8b95ef154a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.423460 4793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d668a84475064a3eaa08b0c719471c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.423669 5807 raft_consensus.cc:491] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:43.424124 5807 raft_consensus.cc:513] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.425990 5807 leader_election.cc:290] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:43.428325 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6740fcfd3d41487b8a4cb2a6af1e70d4" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:43.428867 4819 raft_consensus.cc:2466] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:43.429904 4911 leader_election.cc:304] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:43.430603 5794 raft_consensus.cc:2802] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:43.430984 5794 raft_consensus.cc:491] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:43.431284 5794 raft_consensus.cc:3058] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:43.441277 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8be634cb36d40dca3f9338dab93ad2c" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:43.441498 5798 raft_consensus.cc:491] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:43.442005 5798 raft_consensus.cc:513] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.447566 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6740fcfd3d41487b8a4cb2a6af1e70d4" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:43.448187 4671 raft_consensus.cc:2466] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:43.452162 5026 heartbeater.cc:344] Connected to a master server at 127.4.46.190:33823
I20250626 19:57:43.454636 5807 raft_consensus.cc:491] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:43.455134 5807 raft_consensus.cc:513] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:43.457607 5807 leader_election.cc:290] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:43.471114 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "609d36d46bb84017b2a3a7ad48bfccfc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:43.471737 4819 raft_consensus.cc:2466] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:43.472208 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:43.472906 4911 leader_election.cc:304] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:43.483249 5410 tablet_bootstrap.cc:700] T 5bf4c4596b4441839c82b5fe5c5eea37 P 4d3be519a0044a2f921a503412f72e86: Time spent opening tablet: real 0.117s user 0.004s sys 0.000s
I20250626 19:57:43.487557 5798 leader_election.cc:290] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:43.488101 5798 raft_consensus.cc:2802] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:43.488391 5798 raft_consensus.cc:491] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:43.488667 5798 raft_consensus.cc:3058] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:43.504261 5410 tablet_bootstrap.cc:654] T 5bf4c4596b4441839c82b5fe5c5eea37 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:43.507076 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bf4c4596b4441839c82b5fe5c5eea37" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:43.507722 4819 raft_consensus.cc:2466] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:43.508737 4911 leader_election.cc:304] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:43.509470 5851 raft_consensus.cc:2802] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:43.509768 5851 raft_consensus.cc:491] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:43.510064 5851 raft_consensus.cc:3058] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:43.512653 5802 consensus_queue.cc:1035] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:43.516306 5851 raft_consensus.cc:513] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:43.519073 4756 leader_election.cc:343] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:43.519598 4756 leader_election.cc:304] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:43.520802 5772 raft_consensus.cc:2747] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:43.525504 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "609d36d46bb84017b2a3a7ad48bfccfc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:43.538184 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bf4c4596b4441839c82b5fe5c5eea37" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:43.546586 5366 tablet_bootstrap.cc:654] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:43.554224 4385 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e0bcbb044d1549c88e5b83eeb5c709af consensus state has no leader
I20250626 19:57:43.326315 4942 tablet_service.cc:1468] Processing CreateTablet for tablet 54e68eba747e4a2a947e84e53a444d08 (DEFAULT_TABLE table=default.Table_8ef072e9d240402397d79db39729e650 [id=211696a0b141485ab4e4e2b1d190baa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:43.558604 4942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54e68eba747e4a2a947e84e53a444d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:43.564429 5924 consensus_queue.cc:1035] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:43.571314 5924 consensus_queue.cc:1035] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:43.578327 5851 leader_election.cc:290] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:43.579799 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bf4c4596b4441839c82b5fe5c5eea37" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:43.580364 4817 raft_consensus.cc:3058] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:43.583662 5907 log.cc:826] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Log is configured to *not* fsync() on all Append() calls
I20250626 19:57:43.586875 4817 raft_consensus.cc:2466] T 5bf4c4596b4441839c82b5fe5c5eea37 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:43.589825 4870 heartbeater.cc:344] Connected to a master server at 127.4.46.190:33823
I20250626 19:57:43.597607 4727 heartbeater.cc:344] Connected to a master server at 127.4.46.190:33823
I20250626 19:57:43.611263 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:43.611941 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:43.613246 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:43.615235 5366 tablet_bootstrap.cc:492] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:43.615689 5366 ts_tablet_manager.cc:1397] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.244s user 0.005s sys 0.004s
W20250626 19:57:43.548135 4912 leader_election.cc:343] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:43.588066 4911 leader_election.cc:304] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
W20250626 19:57:43.620389 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 72329.5us
W20250626 19:57:43.620958 4912 leader_election.cc:343] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:43.544844 5794 raft_consensus.cc:513] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.624897 5807 raft_consensus.cc:2802] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:43.625415 5807 raft_consensus.cc:695] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:43.626102 5807 consensus_queue.cc:237] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.629101 5410 tablet_bootstrap.cc:492] T 5bf4c4596b4441839c82b5fe5c5eea37 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:43.321985 5353 ts_tablet_manager.cc:1428] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.099s user 0.006s sys 0.000s
I20250626 19:57:43.629663 5410 ts_tablet_manager.cc:1397] T 5bf4c4596b4441839c82b5fe5c5eea37 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.267s user 0.009s sys 0.000s
I20250626 19:57:43.630457 5353 tablet_bootstrap.cc:492] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:43.632524 5410 raft_consensus.cc:357] T 5bf4c4596b4441839c82b5fe5c5eea37 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.633219 5410 raft_consensus.cc:383] T 5bf4c4596b4441839c82b5fe5c5eea37 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:43.633489 5410 raft_consensus.cc:738] T 5bf4c4596b4441839c82b5fe5c5eea37 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:43.634191 5410 consensus_queue.cc:260] T 5bf4c4596b4441839c82b5fe5c5eea37 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.639097 5794 leader_election.cc:290] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:43.644352 5366 raft_consensus.cc:357] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.646097 5366 raft_consensus.cc:383] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:43.646377 5366 raft_consensus.cc:738] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:43.647065 5366 consensus_queue.cc:260] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.649091 4395 catalog_manager.cc:5582] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:43.644804 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bf4c4596b4441839c82b5fe5c5eea37" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:43.664793 4671 raft_consensus.cc:3058] T 5bf4c4596b4441839c82b5fe5c5eea37 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:43.667080 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6740fcfd3d41487b8a4cb2a6af1e70d4" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:43.667749 4817 raft_consensus.cc:3058] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:43.681005 5798 raft_consensus.cc:513] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:43.690973 4671 raft_consensus.cc:2466] T 5bf4c4596b4441839c82b5fe5c5eea37 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:43.696537 4399 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194] alter complete (version 2)
I20250626 19:57:43.721652 5410 ts_tablet_manager.cc:1428] T 5bf4c4596b4441839c82b5fe5c5eea37 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.092s user 0.006s sys 0.001s
I20250626 19:57:43.723268 5410 tablet_bootstrap.cc:492] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
W20250626 19:57:43.725823 4392 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 consensus state has no leader
I20250626 19:57:43.726495 4392 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34796 (request call id 130) took 4342 ms. Trace:
I20250626 19:57:43.726801 4392 rpcz_store.cc:276] 0626 19:57:39.384040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.649989 (+265949us) service_pool.cc:225] Handling call
0626 19:57:39.650815 (+ 826us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:39.651500 (+ 685us) catalog_manager.cc:3683] Apply alter schema
0626 19:57:39.652030 (+ 530us) catalog_manager.cc:3715] Acquired catalog manager lock
0626 19:57:39.652212 (+ 182us) catalog_manager.cc:3873] Updating metadata on disk
0626 19:57:39.780094 (+127882us) catalog_manager.cc:3904] Committing alterations to in-memory state
0626 19:57:43.726379 (+3946285us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:39.654289 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:39.654626 (+ 337us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:39.654646 (+ 20us) write_op.cc:435] Acquired schema lock
0626 19:57:39.654657 (+ 11us) tablet.cc:660] Decoding operations
0626 19:57:39.654892 (+ 235us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:39.654925 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:39.654945 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:39.655065 (+ 120us) tablet.cc:699] Row locks acquired
0626 19:57:39.655077 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:39.655170 (+ 93us) write_op.cc:270] Start()
0626 19:57:39.655237 (+ 67us) write_op.cc:276] Timestamp: P: 1750967859655198 usec, L: 0
0626 19:57:39.655251 (+ 14us) op_driver.cc:348] REPLICATION: starting
0626 19:57:39.655679 (+ 428us) log.cc:844] Serialized 405 byte log entry
0626 19:57:39.769130 (+113451us) op_driver.cc:464] REPLICATION: finished
0626 19:57:39.778395 (+ 9265us) write_op.cc:301] APPLY: starting
0626 19:57:39.778447 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:39.778532 (+ 85us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:39.778637 (+ 105us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:39.778794 (+ 157us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:39.779004 (+ 210us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:39.779048 (+ 44us) write_op.cc:312] APPLY: finished
0626 19:57:39.779222 (+ 174us) log.cc:844] Serialized 23 byte log entry
0626 19:57:39.779824 (+ 602us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:39.779883 (+ 59us) write_op.cc:454] Released schema lock
0626 19:57:39.779912 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":9126,"apply.run_cpu_time_us":1189,"apply.run_wall_time_us":1975,"num_ops":1,"prepare.queue_time_us":99,"prepare.run_cpu_time_us":1591,"prepare.run_wall_time_us":2617,"replication_time_us":113807}]]}
W20250626 19:57:43.733163 4392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 75) took 2282 ms (client timeout 1000 ms). Trace:
W20250626 19:57:43.733484 4392 rpcz_store.cc:269] 0626 19:57:41.450285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:43.732809 (+2282524us) service_pool.cc:212] Skipping call since client already timed out
0626 19:57:43.733096 (+ 287us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250626 19:57:43.736626 4392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 81) took 1603 ms (client timeout 1000 ms). Trace:
W20250626 19:57:43.736946 4392 rpcz_store.cc:269] 0626 19:57:42.130717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:43.734331 (+1603614us) service_pool.cc:212] Skipping call since client already timed out
0626 19:57:43.734457 (+ 126us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250626 19:57:43.738816 4392 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.46.131:45585 (request call id 67) took 1606 ms (client timeout 1000 ms). Trace:
W20250626 19:57:43.739095 4392 rpcz_store.cc:269] 0626 19:57:42.131261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:43.737662 (+1606401us) service_pool.cc:212] Skipping call since client already timed out
0626 19:57:43.737767 (+ 105us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250626 19:57:43.739876 4392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 76) took 1211 ms (client timeout 1000 ms). Trace:
W20250626 19:57:43.740152 4392 rpcz_store.cc:269] 0626 19:57:42.528936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:43.739623 (+1210687us) service_pool.cc:212] Skipping call since client already timed out
0626 19:57:43.739829 (+ 206us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250626 19:57:43.749032 4817 raft_consensus.cc:2466] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:43.758249 4393 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34796 (request call id 129) took 4380 ms. Trace:
I20250626 19:57:43.758565 4393 rpcz_store.cc:276] 0626 19:57:39.377525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.644880 (+267355us) service_pool.cc:225] Handling call
0626 19:57:39.645792 (+ 912us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:39.647503 (+ 1711us) catalog_manager.cc:3683] Apply alter schema
0626 19:57:39.648144 (+ 641us) catalog_manager.cc:3715] Acquired catalog manager lock
0626 19:57:39.648349 (+ 205us) catalog_manager.cc:3873] Updating metadata on disk
0626 19:57:39.820745 (+172396us) catalog_manager.cc:3904] Committing alterations to in-memory state
0626 19:57:43.758083 (+3937338us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:39.685741 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:39.686134 (+ 393us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:39.686157 (+ 23us) write_op.cc:435] Acquired schema lock
0626 19:57:39.686169 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:39.686398 (+ 229us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:39.686431 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:39.686452 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:39.686578 (+ 126us) tablet.cc:699] Row locks acquired
0626 19:57:39.686591 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:57:39.686710 (+ 119us) write_op.cc:270] Start()
0626 19:57:39.686818 (+ 108us) write_op.cc:276] Timestamp: P: 1750967859686747 usec, L: 0
0626 19:57:39.686841 (+ 23us) op_driver.cc:348] REPLICATION: starting
0626 19:57:39.799131 (+112290us) log.cc:844] Serialized 405 byte log entry
0626 19:57:39.801973 (+ 2842us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72b481 00007fe78f7a2a18 00007fe78fc0f811 00007fe78fc0efe8 00007fe78fc13c81 00007fe78fc13c39 00007fe78fc13bc9 00007fe78fc13b91 00007fe78fc12dbd 000000000059bcd4 000000000059bb09
0626 19:57:39.808985 (+ 7012us) op_driver.cc:464] REPLICATION: finished
0626 19:57:39.818028 (+ 9043us) write_op.cc:301] APPLY: starting
0626 19:57:39.818081 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:39.818172 (+ 91us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:39.818191 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:39.818341 (+ 150us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:39.818455 (+ 114us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:39.818496 (+ 41us) write_op.cc:312] APPLY: finished
0626 19:57:39.819789 (+ 1293us) log.cc:844] Serialized 23 byte log entry
0626 19:57:39.820411 (+ 622us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:39.820468 (+ 57us) write_op.cc:454] Released schema lock
0626 19:57:39.820505 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:39.821080 (+ 575us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:39.821106 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":8872,"apply.run_cpu_time_us":1234,"apply.run_wall_time_us":3031,"num_ops":1,"prepare.queue_time_us":6171,"prepare.run_cpu_time_us":4586,"prepare.run_wall_time_us":119844,"raft.queue_time_us":963,"raft.run_cpu_time_us":666,"raft.run_wall_time_us":667,"replication_time_us":122064,"spinlock_wait_cycles":257189760}]]}
I20250626 19:57:43.772392 4869 heartbeater.cc:344] Connected to a master server at 127.4.46.189:45039
I20250626 19:57:43.772867 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:43.772306 4393 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_bf54a23958c14a1699f712665f29c0ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:43.782160 5353 tablet_bootstrap.cc:654] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:43.789866 4393 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf54a23958c14a1699f712665f29c0ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:43.801867 5366 ts_tablet_manager.cc:1428] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.186s user 0.006s sys 0.000s
I20250626 19:57:43.803577 4397 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 609d36d46bb84017b2a3a7ad48bfccfc
I20250626 19:57:43.804953 4397 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34796 (request call id 127) took 4434 ms. Trace:
I20250626 19:57:43.808488 4794 tablet_service.cc:1515] Processing DeleteTablet for tablet 609d36d46bb84017b2a3a7ad48bfccfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:43.805225 4397 rpcz_store.cc:276] 0626 19:57:39.370930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.562425 (+191495us) service_pool.cc:225] Handling call
0626 19:57:39.649471 (+ 87046us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:39.731256 (+ 81785us) catalog_manager.cc:2790] Modifying in-memory table state
0626 19:57:39.731423 (+ 167us) catalog_manager.cc:2799] Locking tablets
0626 19:57:40.500571 (+769148us) catalog_manager.cc:2813] Removing table and tablets from system table
0626 19:57:40.543425 (+ 42854us) catalog_manager.cc:2831] Removing table from by-name map
0626 19:57:40.543687 (+ 262us) catalog_manager.cc:2847] Committing in-memory state
0626 19:57:43.803380 (+3259693us) catalog_manager.cc:2851] Aborting table tasks
0626 19:57:43.804877 (+ 1497us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:40.530986 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:40.531590 (+ 604us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:40.531625 (+ 35us) write_op.cc:435] Acquired schema lock
0626 19:57:40.531646 (+ 21us) tablet.cc:660] Decoding operations
0626 19:57:40.532049 (+ 403us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:40.532104 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:40.532137 (+ 33us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:40.532456 (+ 319us) tablet.cc:699] Row locks acquired
0626 19:57:40.532476 (+ 20us) write_op.cc:260] PREPARE: finished
0626 19:57:40.532633 (+ 157us) write_op.cc:270] Start()
0626 19:57:40.532742 (+ 109us) write_op.cc:276] Timestamp: P: 1750967860532774 usec, L: 0
0626 19:57:40.532765 (+ 23us) op_driver.cc:348] REPLICATION: starting
0626 19:57:40.533426 (+ 661us) log.cc:844] Serialized 728 byte log entry
0626 19:57:40.541222 (+ 7796us) op_driver.cc:464] REPLICATION: finished
0626 19:57:40.541749 (+ 527us) write_op.cc:301] APPLY: starting
0626 19:57:40.541817 (+ 68us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:40.541962 (+ 145us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:40.542072 (+ 110us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:40.542360 (+ 288us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:40.542559 (+ 199us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:40.542622 (+ 63us) write_op.cc:312] APPLY: finished
0626 19:57:40.542880 (+ 258us) log.cc:844] Serialized 29 byte log entry
0626 19:57:40.543053 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:40.543144 (+ 91us) write_op.cc:454] Released schema lock
0626 19:57:40.543186 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:40.557324 (+ 14138us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:40.557363 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":358,"apply.run_cpu_time_us":1693,"apply.run_wall_time_us":2125,"num_ops":2,"prepare.queue_time_us":28685,"prepare.run_cpu_time_us":2746,"prepare.run_wall_time_us":26334,"raft.queue_time_us":84,"raft.run_cpu_time_us":560,"raft.run_wall_time_us":562,"replication_time_us":8346}]]}
I20250626 19:57:43.810973 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet 609d36d46bb84017b2a3a7ad48bfccfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:43.812109 5366 tablet_bootstrap.cc:492] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:43.815634 5936 tablet_replica.cc:331] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:43.815753 5887 raft_consensus.cc:491] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:43.816197 5887 raft_consensus.cc:513] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.818447 5887 leader_election.cc:290] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:43.818845 5936 raft_consensus.cc:2241] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:43.818814 4397 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_c46f810d71934cdeb268c25bc98d2325"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:43.819350 5936 raft_consensus.cc:2270] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:43.819844 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "781f8ee8c8ad49c6b2391fcdf20bdfdb" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:43.820408 4820 raft_consensus.cc:2466] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
W20250626 19:57:43.820771 4397 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c46f810d71934cdeb268c25bc98d2325 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:43.821163 5936 ts_tablet_manager.cc:1905] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:43.821302 4609 leader_election.cc:304] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:43.822160 5887 raft_consensus.cc:2802] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:43.822495 5887 raft_consensus.cc:491] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:43.822878 5887 raft_consensus.cc:3058] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:43.825493 5506 raft_consensus.cc:491] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:43.828401 5506 raft_consensus.cc:513] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.829977 5506 leader_election.cc:290] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:43.831686 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9172cb6576b240f6b25c3b1267618e6c" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:43.833009 4755 leader_election.cc:343] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:43.845755 5798 leader_election.cc:290] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:43.860466 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 609d36d46bb84017b2a3a7ad48bfccfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 609d36d46bb84017b2a3a7ad48bfccfc already in progress: creating tablet
I20250626 19:57:43.867719 5366 tablet_bootstrap.cc:654] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:43.873771 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 94327cab01df4338aaec157736d106f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:36 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:43.873040 4794 tablet_service.cc:1515] Processing DeleteTablet for tablet f43aed8f07844c5ca727ae6df63896df with delete_type TABLET_DATA_DELETED (replaced by tablet 075dbe46bab14d40a8008ce5c24bcf38) from {username='slave'} at 127.0.0.1:40742
W20250626 19:57:43.874176 4383 catalog_manager.cc:5001] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): alter failed for tablet 6b66cda4f8ed410184b434d5083deac4 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]),no further retry: Not found: Tablet not found: 6b66cda4f8ed410184b434d5083deac4
I20250626 19:57:43.874487 4793 tablet_service.cc:1515] Processing DeleteTablet for tablet 82823fd9ceb1485386616319fa7aeb43 with delete_type TABLET_DATA_DELETED (replaced by tablet f6ec952321924098a142f1438a283aab) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:43.875638 4793 tablet_service.cc:1515] Processing DeleteTablet for tablet 7391b9eaea3e426ba4118fc8b53b72a6 with delete_type TABLET_DATA_DELETED (replaced by tablet d8399c219d0e4485bb7926eefd78f210) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:43.883404 5936 ts_tablet_manager.cc:1918] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:43.888231 4794 tablet_service.cc:1515] Processing DeleteTablet for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a6058961bc7450ab489b26158e7dd17) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:43.889271 5936 log.cc:1199] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/609d36d46bb84017b2a3a7ad48bfccfc
I20250626 19:57:43.890071 5936 ts_tablet_manager.cc:1939] T 609d36d46bb84017b2a3a7ad48bfccfc P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:43.894454 4793 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b66cda4f8ed410184b434d5083deac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7851e464fcd44ecb82404d6a991db2a2) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:43.895342 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b8473259485460783f62f8193d5b02c with delete_type TABLET_DATA_DELETED (replaced by tablet 8282e22bfb8b4d4994fa8b95ef154a68) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:43.895840 4800 tablet_service.cc:1515] Processing DeleteTablet for tablet 36997dab0ab3430bba63a90805759b30 with delete_type TABLET_DATA_DELETED (replaced by tablet fafbf98e7f4544b6ad96c1c5ab6c82bc) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:43.898437 4651 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c03691d021e4ee19d5bb287ab633a3c with delete_type TABLET_DATA_DELETED (replaced by tablet de0724ba71824a5daaf6f3708b4431b7) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:43.898698 4648 tablet_service.cc:1515] Processing DeleteTablet for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a6058961bc7450ab489b26158e7dd17) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:43.899814 4800 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c03691d021e4ee19d5bb287ab633a3c with delete_type TABLET_DATA_DELETED (replaced by tablet de0724ba71824a5daaf6f3708b4431b7) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:43.900115 4654 tablet_service.cc:1515] Processing DeleteTablet for tablet 82823fd9ceb1485386616319fa7aeb43 with delete_type TABLET_DATA_DELETED (replaced by tablet f6ec952321924098a142f1438a283aab) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:43.900318 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 94cefa59de044a36a2afaaf5531128d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:43.900614 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b66cda4f8ed410184b434d5083deac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7851e464fcd44ecb82404d6a991db2a2) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:43.901569 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:43.901638 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet 94327cab01df4338aaec157736d106f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:36 UTC) from {username='slave'} at 127.0.0.1:49550
W20250626 19:57:43.902067 4394 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 73) took 5166 ms (client timeout 1000 ms). Trace:
I20250626 19:57:43.902570 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 609d36d46bb84017b2a3a7ad48bfccfc (table default.Table_d23ee99af7b845b78d2200b962f681a6 [id=fc725d6df6774d4e922e2d98d461b111]) successfully deleted
W20250626 19:57:43.903705 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 94327cab01df4338aaec157736d106f6 because the tablet was not found. No further retry: Not found: Tablet not found: 94327cab01df4338aaec157736d106f6
W20250626 19:57:43.904486 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet f43aed8f07844c5ca727ae6df63896df because the tablet was not found. No further retry: Not found: Tablet not found: f43aed8f07844c5ca727ae6df63896df
I20250626 19:57:43.905030 4649 tablet_service.cc:1515] Processing DeleteTablet for tablet f43aed8f07844c5ca727ae6df63896df with delete_type TABLET_DATA_DELETED (replaced by tablet 075dbe46bab14d40a8008ce5c24bcf38) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:43.906906 5802 consensus_queue.cc:1035] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:43.908550 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 609d36d46bb84017b2a3a7ad48bfccfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:40742
W20250626 19:57:43.909801 4390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 74) took 3851 ms (client timeout 1000 ms). Trace:
W20250626 19:57:43.911567 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 3c03691d021e4ee19d5bb287ab633a3c because the tablet was not found. No further retry: Not found: Tablet not found: 3c03691d021e4ee19d5bb287ab633a3c
W20250626 19:57:43.912518 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 82823fd9ceb1485386616319fa7aeb43 because the tablet was not found. No further retry: Not found: Tablet not found: 82823fd9ceb1485386616319fa7aeb43
W20250626 19:57:43.913004 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 because the tablet was not found. No further retry: Not found: Tablet not found: dd0ca2fab40f45e5a1f9bd6059ca1f08
W20250626 19:57:43.913214 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 7391b9eaea3e426ba4118fc8b53b72a6 because the tablet was not found. No further retry: Not found: Tablet not found: 7391b9eaea3e426ba4118fc8b53b72a6
W20250626 19:57:43.913854 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 because the tablet was not found. No further retry: Not found: Tablet not found: dd0ca2fab40f45e5a1f9bd6059ca1f08
W20250626 19:57:43.914364 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 82823fd9ceb1485386616319fa7aeb43 because the tablet was not found. No further retry: Not found: Tablet not found: 82823fd9ceb1485386616319fa7aeb43
W20250626 19:57:43.914407 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 6b66cda4f8ed410184b434d5083deac4 because the tablet was not found. No further retry: Not found: Tablet not found: 6b66cda4f8ed410184b434d5083deac4
W20250626 19:57:43.915409 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 6b8473259485460783f62f8193d5b02c because the tablet was not found. No further retry: Not found: Tablet not found: 6b8473259485460783f62f8193d5b02c
W20250626 19:57:43.915714 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 6b66cda4f8ed410184b434d5083deac4 because the tablet was not found. No further retry: Not found: Tablet not found: 6b66cda4f8ed410184b434d5083deac4
W20250626 19:57:43.916759 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 36997dab0ab3430bba63a90805759b30 because the tablet was not found. No further retry: Not found: Tablet not found: 36997dab0ab3430bba63a90805759b30
I20250626 19:57:43.918080 5936 tablet_replica.cc:331] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
W20250626 19:57:43.920002 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 3c03691d021e4ee19d5bb287ab633a3c because the tablet was not found. No further retry: Not found: Tablet not found: 3c03691d021e4ee19d5bb287ab633a3c
I20250626 19:57:43.923477 5936 raft_consensus.cc:2241] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250626 19:57:43.902390 4394 rpcz_store.cc:269] 0626 19:57:38.735604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.170013 (+434409us) service_pool.cc:225] Handling call
0626 19:57:43.901871 (+4731858us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:40.474854 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:40.475254 (+ 400us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:40.475279 (+ 25us) write_op.cc:435] Acquired schema lock
0626 19:57:40.475291 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:40.475589 (+ 298us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:40.475628 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:40.475650 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:40.475876 (+ 226us) tablet.cc:699] Row locks acquired
0626 19:57:40.475888 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:40.475996 (+ 108us) write_op.cc:270] Start()
0626 19:57:40.476076 (+ 80us) write_op.cc:276] Timestamp: P: 1750967860476126 usec, L: 0
0626 19:57:40.476093 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:40.476721 (+ 628us) log.cc:844] Serialized 907 byte log entry
0626 19:57:40.497686 (+ 20965us) op_driver.cc:464] REPLICATION: finished
0626 19:57:40.497999 (+ 313us) write_op.cc:301] APPLY: starting
0626 19:57:40.498052 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:40.498148 (+ 96us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:40.498234 (+ 86us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:40.498452 (+ 218us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:40.498590 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:40.498633 (+ 43us) write_op.cc:312] APPLY: finished
0626 19:57:40.498823 (+ 190us) log.cc:844] Serialized 29 byte log entry
0626 19:57:40.498953 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:40.499018 (+ 65us) write_op.cc:454] Released schema lock
0626 19:57:40.499047 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:40.499255 (+ 208us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:40.499277 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":134464,"reported_tablets":52,"child_traces":[["op",{"apply.queue_time_us":188,"apply.run_cpu_time_us":1219,"apply.run_wall_time_us":1217,"num_ops":2,"prepare.queue_time_us":44654,"prepare.run_cpu_time_us":2074,"prepare.run_wall_time_us":20628,"raft.queue_time_us":104,"raft.run_cpu_time_us":1277,"raft.run_wall_time_us":1276,"replication_time_us":21509}]]}
W20250626 19:57:43.923238 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 94cefa59de044a36a2afaaf5531128d8 because the tablet was not found. No further retry: Not found: Tablet not found: 94cefa59de044a36a2afaaf5531128d8
I20250626 19:57:43.926290 5410 tablet_bootstrap.cc:654] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:43.929483 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 94327cab01df4338aaec157736d106f6 because the tablet was not found. No further retry: Not found: Tablet not found: 94327cab01df4338aaec157736d106f6
I20250626 19:57:43.930388 5936 raft_consensus.cc:2270] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 19:57:43.930922 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet f43aed8f07844c5ca727ae6df63896df because the tablet was not found. No further retry: Not found: Tablet not found: f43aed8f07844c5ca727ae6df63896df
W20250626 19:57:43.933302 4383 catalog_manager.cc:5001] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): alter failed for tablet 6b66cda4f8ed410184b434d5083deac4 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]),no further retry: Not found: Tablet not found: 6b66cda4f8ed410184b434d5083deac4
I20250626 19:57:43.929949 5366 tablet_bootstrap.cc:492] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:43.934019 5366 ts_tablet_manager.cc:1397] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.122s user 0.017s sys 0.000s
I20250626 19:57:43.936347 5366 raft_consensus.cc:357] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.936961 5366 raft_consensus.cc:383] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:43.937225 5366 raft_consensus.cc:738] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:43.939595 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 609d36d46bb84017b2a3a7ad48bfccfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:43.940279 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:43.941072 5366 consensus_queue.cc:260] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.943298 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b8473259485460783f62f8193d5b02c with delete_type TABLET_DATA_DELETED (replaced by tablet 8282e22bfb8b4d4994fa8b95ef154a68) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:43.943955 4649 tablet_service.cc:1515] Processing DeleteTablet for tablet 36997dab0ab3430bba63a90805759b30 with delete_type TABLET_DATA_DELETED (replaced by tablet fafbf98e7f4544b6ad96c1c5ab6c82bc) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:43.932457 5936 ts_tablet_manager.cc:1905] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:43.950736 5942 tablet.cc:1722] T 2fda659c30c44c8ba402ac5c7f3598e0 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:43.956792 5938 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 62ac80df82774b9b87c0be076b5637af)
I20250626 19:57:43.957384 5938 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:43.947583 5506 raft_consensus.cc:491] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:43.958992 5506 raft_consensus.cc:513] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:43.961020 5506 leader_election.cc:290] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:43.961970 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "125f7265258e48368381aa54559eb510" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:43.963338 4755 leader_election.cc:343] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:43.964171 4650 tablet_service.cc:1515] Processing DeleteTablet for tablet 94cefa59de044a36a2afaaf5531128d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:49550
W20250626 19:57:43.910096 4390 rpcz_store.cc:269] 0626 19:57:40.058319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:40.058579 (+ 260us) service_pool.cc:225] Handling call
0626 19:57:40.074312 (+ 15733us) spinlock_profiling.cc:243] Waited 4.14 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe7915557ee 00007fe78f26c624 00007fe78f26c5b2 00007fe78f26c4e1 00007fe78f26c45c 00007fe78f26b712 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:43.909603 (+3835291us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:42.696330 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:42.696709 (+ 379us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:42.696736 (+ 27us) write_op.cc:435] Acquired schema lock
0626 19:57:42.696749 (+ 13us) tablet.cc:660] Decoding operations
0626 19:57:42.697008 (+ 259us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:42.697046 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:42.697068 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:42.697197 (+ 129us) tablet.cc:699] Row locks acquired
0626 19:57:42.697209 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:42.697315 (+ 106us) write_op.cc:270] Start()
0626 19:57:42.697399 (+ 84us) write_op.cc:276] Timestamp: P: 1750967862697322 usec, L: 0
0626 19:57:42.697415 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:57:42.697916 (+ 501us) log.cc:844] Serialized 530 byte log entry
0626 19:57:42.703020 (+ 5104us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b1800041020. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe791479af1 00007fe791486441 00007fe7888293e6 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe791479af1 00007fe791486441 00007fe7888293e6 00007fe788829380
0626 19:57:42.770412 (+ 67392us) op_driver.cc:464] REPLICATION: finished
0626 19:57:42.788402 (+ 17990us) write_op.cc:301] APPLY: starting
0626 19:57:42.788467 (+ 65us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:42.788549 (+ 82us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:42.788569 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:42.788743 (+ 174us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:42.788884 (+ 141us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:42.788933 (+ 49us) write_op.cc:312] APPLY: finished
0626 19:57:42.789113 (+ 180us) log.cc:844] Serialized 23 byte log entry
0626 19:57:42.791937 (+ 2824us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:42.792003 (+ 66us) write_op.cc:454] Released schema lock
0626 19:57:42.792036 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:42.824167 (+ 32131us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:42.824195 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":83475,"reported_tablets":55,"spinlock_wait_cycles":9532160,"child_traces":[["op",{"apply.queue_time_us":17806,"apply.run_cpu_time_us":1257,"apply.run_wall_time_us":34650,"num_ops":1,"prepare.queue_time_us":55758,"prepare.run_cpu_time_us":1916,"prepare.run_wall_time_us":1916,"raft.queue_time_us":1626,"raft.run_cpu_time_us":700,"raft.run_wall_time_us":702,"replication_time_us":72894,"spinlock_wait_cycles":5419520,"thread_start_us":17913,"threads_started":1}]]}
I20250626 19:57:43.946823 5366 ts_tablet_manager.cc:1428] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250626 19:57:43.971606 5366 tablet_bootstrap.cc:492] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:43.971618 4394 master_service.cc:948] ReplaceTablet: received request to replace tablet 4c0338c05a974e0f96865398722c5e8f from {username='slave'} at 127.0.0.1:34796
W20250626 19:57:43.972841 4396 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 64) took 5201 ms (client timeout 1000 ms). Trace:
I20250626 19:57:43.973913 5938 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:43.991794 4965 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "125f7265258e48368381aa54559eb510" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:43.999657 4969 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9172cb6576b240f6b25c3b1267618e6c" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:44.004973 4972 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "781f8ee8c8ad49c6b2391fcdf20bdfdb" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:43.976497 4407 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 7 candidate_status { last_received { term: 6 index: 226 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:44.020112 5947 tablet.cc:1722] T 2fda659c30c44c8ba402ac5c7f3598e0 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250626 19:57:44.021152 4396 rpcz_store.cc:269] 0626 19:57:38.770879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.170503 (+399624us) service_pool.cc:225] Handling call
0626 19:57:43.972674 (+4802171us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:41.299181 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:41.299607 (+ 426us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:41.299631 (+ 24us) write_op.cc:435] Acquired schema lock
0626 19:57:41.299644 (+ 13us) tablet.cc:660] Decoding operations
0626 19:57:41.299879 (+ 235us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:41.299919 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:41.299946 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:41.300117 (+ 171us) tablet.cc:699] Row locks acquired
0626 19:57:41.300138 (+ 21us) write_op.cc:260] PREPARE: finished
0626 19:57:41.300250 (+ 112us) write_op.cc:270] Start()
0626 19:57:41.300347 (+ 97us) write_op.cc:276] Timestamp: P: 1750967861300314 usec, L: 0
0626 19:57:41.300370 (+ 23us) op_driver.cc:348] REPLICATION: starting
0626 19:57:41.300891 (+ 521us) log.cc:844] Serialized 530 byte log entry
0626 19:57:41.316020 (+ 15129us) op_driver.cc:464] REPLICATION: finished
0626 19:57:41.316473 (+ 453us) write_op.cc:301] APPLY: starting
0626 19:57:41.316543 (+ 70us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:41.316660 (+ 117us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:41.316778 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:41.316988 (+ 210us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:41.317155 (+ 167us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:41.317219 (+ 64us) write_op.cc:312] APPLY: finished
0626 19:57:41.317429 (+ 210us) log.cc:844] Serialized 23 byte log entry
0626 19:57:41.317576 (+ 147us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:41.317639 (+ 63us) write_op.cc:454] Released schema lock
0626 19:57:41.317669 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:41.317892 (+ 223us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:41.317912 (+ 20us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":120244,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":287,"apply.run_cpu_time_us":1382,"apply.run_wall_time_us":1380,"num_ops":1,"prepare.queue_time_us":88527,"prepare.run_cpu_time_us":2105,"prepare.run_wall_time_us":2105,"raft.queue_time_us":9699,"raft.run_cpu_time_us":779,"raft.run_wall_time_us":779,"replication_time_us":15516}]]}
W20250626 19:57:44.032727 4399 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 65) took 4180 ms (client timeout 1000 ms). Trace:
W20250626 19:57:44.040861 4756 leader_election.cc:343] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:44.041276 4756 leader_election.cc:304] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:44.042359 5944 raft_consensus.cc:2747] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:44.043992 5623 raft_consensus.cc:491] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:44.044430 5623 raft_consensus.cc:513] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.046270 5623 leader_election.cc:290] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:44.048470 4649 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b66cda4f8ed410184b434d5083deac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7851e464fcd44ecb82404d6a991db2a2) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:44.049371 4649 tablet_service.cc:1515] Processing DeleteTablet for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a6058961bc7450ab489b26158e7dd17) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:44.051059 4649 tablet_service.cc:1515] Processing DeleteTablet for tablet fc526a05990446a3b3bc973e79511476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:44.053270 4649 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:49550
W20250626 19:57:44.057216 4756 leader_election.cc:343] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:44.058386 4756 leader_election.cc:304] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:44.059352 5944 raft_consensus.cc:2747] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:43.991842 4398 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 79) took 4127 ms (client timeout 1000 ms). Trace:
W20250626 19:57:44.060153 4398 rpcz_store.cc:269] 0626 19:57:39.864467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.923143 (+ 58676us) service_pool.cc:225] Handling call
0626 19:57:40.079328 (+156185us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe7915557ee 00007fe78f26c624 00007fe78f26c5b2 00007fe78f26c4e1 00007fe78f26c45c 00007fe78f26b712 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:43.991542 (+3912214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":74286,"reported_tablets":56,"spinlock_wait_cycles":344246528}
I20250626 19:57:44.062153 4398 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_987278d268e04477828ad87c6d9c8c01" } modify_external_catalogs: true
I20250626 19:57:44.062721 4398 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_987278d268e04477828ad87c6d9c8c01" } modify_external_catalogs: true
W20250626 19:57:44.017438 4391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 80) took 3029 ms (client timeout 1000 ms). Trace:
I20250626 19:57:44.039819 4397 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 143) took 3487 ms. Trace:
I20250626 19:57:44.066275 4397 rpcz_store.cc:276] 0626 19:57:40.552397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:43.811191 (+3258794us) service_pool.cc:225] Handling call
0626 19:57:43.821200 (+ 10009us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:43.822607 (+ 1407us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:44.039132 (+216525us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:44.039471 (+ 339us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:44.039744 (+ 273us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:43.836554 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:43.838774 (+ 2220us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:43.838803 (+ 29us) write_op.cc:435] Acquired schema lock
0626 19:57:43.838818 (+ 15us) tablet.cc:660] Decoding operations
0626 19:57:43.839674 (+ 856us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:43.839715 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:43.839747 (+ 32us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:43.839960 (+ 213us) tablet.cc:699] Row locks acquired
0626 19:57:43.839975 (+ 15us) write_op.cc:260] PREPARE: finished
0626 19:57:43.840097 (+ 122us) write_op.cc:270] Start()
0626 19:57:43.840180 (+ 83us) write_op.cc:276] Timestamp: P: 1750967863840209 usec, L: 0
0626 19:57:43.840200 (+ 20us) op_driver.cc:348] REPLICATION: starting
0626 19:57:43.840722 (+ 522us) log.cc:844] Serialized 433 byte log entry
0626 19:57:44.014047 (+173325us) op_driver.cc:464] REPLICATION: finished
0626 19:57:44.030622 (+ 16575us) write_op.cc:301] APPLY: starting
0626 19:57:44.030691 (+ 69us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:44.033180 (+ 2489us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:44.033296 (+ 116us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:44.036873 (+ 3577us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:44.037027 (+ 154us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:44.037074 (+ 47us) write_op.cc:312] APPLY: finished
0626 19:57:44.037258 (+ 184us) log.cc:844] Serialized 29 byte log entry
0626 19:57:44.038827 (+ 1569us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.038898 (+ 71us) write_op.cc:454] Released schema lock
0626 19:57:44.038935 (+ 37us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":16395,"apply.run_cpu_time_us":1888,"apply.run_wall_time_us":12048,"num_ops":2,"prepare.queue_time_us":12734,"prepare.run_cpu_time_us":2298,"prepare.run_wall_time_us":15016,"replication_time_us":173739,"thread_start_us":29616,"threads_started":1}]]}
I20250626 19:57:44.054642 4650 tablet_service.cc:1515] Processing DeleteTablet for tablet f43aed8f07844c5ca727ae6df63896df with delete_type TABLET_DATA_DELETED (replaced by tablet 075dbe46bab14d40a8008ce5c24bcf38) from {username='slave'} at 127.0.0.1:49550
W20250626 19:57:44.069736 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 609d36d46bb84017b2a3a7ad48bfccfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 609d36d46bb84017b2a3a7ad48bfccfc already in progress: creating tablet
I20250626 19:57:44.069962 5937 tablet_replica.cc:331] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:44.070775 5937 raft_consensus.cc:2241] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:44.071249 5937 raft_consensus.cc:2270] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:44.071130 4397 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_e6347deb7eb14f08bd5e02821a5a55dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:44.072809 4397 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6347deb7eb14f08bd5e02821a5a55dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:44.073037 5937 ts_tablet_manager.cc:1905] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:44.055388 4390 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_6e9e80ead85b44f19cd580a82983fe06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:44.079476 4390 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e9e80ead85b44f19cd580a82983fe06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:57:44.082019 4391 rpcz_store.cc:269] 0626 19:57:40.987911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:40.992444 (+ 4533us) service_pool.cc:225] Handling call
0626 19:57:44.017339 (+3024895us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64577,"reported_tablets":61}
I20250626 19:57:44.083926 4391 master_service.cc:948] ReplaceTablet: received request to replace tablet f6ec952321924098a142f1438a283aab from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:44.076473 5887 raft_consensus.cc:513] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:44.089231 4611 leader_election.cc:343] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:44.092435 5410 tablet_bootstrap.cc:492] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:44.092965 5410 ts_tablet_manager.cc:1397] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.370s user 0.007s sys 0.002s
I20250626 19:57:44.095301 5410 raft_consensus.cc:357] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.095870 4870 heartbeater.cc:344] Connected to a master server at 127.4.46.190:33823
I20250626 19:57:44.102267 5366 tablet_bootstrap.cc:654] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:44.104841 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69bd106066b6492bb7ea03d15a18df00" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:44.105436 4819 raft_consensus.cc:2466] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:44.111912 4393 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 142) took 3616 ms. Trace:
I20250626 19:57:44.112239 4393 rpcz_store.cc:276] 0626 19:57:40.495321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:43.771895 (+3276574us) service_pool.cc:225] Handling call
0626 19:57:43.790498 (+ 18603us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:43.838216 (+ 47718us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:44.111210 (+272994us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:44.111584 (+ 374us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:44.111837 (+ 253us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:43.914974 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:43.915427 (+ 453us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:43.915462 (+ 35us) write_op.cc:435] Acquired schema lock
0626 19:57:43.915478 (+ 16us) tablet.cc:660] Decoding operations
0626 19:57:43.915803 (+ 325us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:43.915847 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:43.915903 (+ 56us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:43.916120 (+ 217us) tablet.cc:699] Row locks acquired
0626 19:57:43.916138 (+ 18us) write_op.cc:260] PREPARE: finished
0626 19:57:43.916258 (+ 120us) write_op.cc:270] Start()
0626 19:57:43.916351 (+ 93us) write_op.cc:276] Timestamp: P: 1750967863916396 usec, L: 0
0626 19:57:43.916371 (+ 20us) op_driver.cc:348] REPLICATION: starting
0626 19:57:43.916944 (+ 573us) log.cc:844] Serialized 433 byte log entry
0626 19:57:44.052361 (+135417us) op_driver.cc:464] REPLICATION: finished
0626 19:57:44.058333 (+ 5972us) write_op.cc:301] APPLY: starting
0626 19:57:44.058395 (+ 62us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:44.058495 (+ 100us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:44.058582 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:44.059005 (+ 423us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:44.059137 (+ 132us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:44.059178 (+ 41us) write_op.cc:312] APPLY: finished
0626 19:57:44.059351 (+ 173us) log.cc:844] Serialized 29 byte log entry
0626 19:57:44.110913 (+ 51562us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.110985 (+ 72us) write_op.cc:454] Released schema lock
0626 19:57:44.111019 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5801,"num_ops":2,"prepare.queue_time_us":71863,"prepare.run_cpu_time_us":2168,"prepare.run_wall_time_us":3134,"replication_time_us":135885}]]}
I20250626 19:57:44.116876 4393 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_fde5b6bd4ee8417e85ffccad3bd2a581" } new_table_name: "default.Table_5fe142a9bd464f8ea2a070dda8bb61e9" modify_external_catalogs: true
I20250626 19:57:44.027530 4911 leader_election.cc:304] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:44.130048 4952 tablet_service.cc:1515] Processing DeleteTablet for tablet 609d36d46bb84017b2a3a7ad48bfccfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:44.130333 4950 tablet_service.cc:1515] Processing DeleteTablet for tablet f43aed8f07844c5ca727ae6df63896df with delete_type TABLET_DATA_DELETED (replaced by tablet 075dbe46bab14d40a8008ce5c24bcf38) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:44.130734 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 94327cab01df4338aaec157736d106f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:36 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:44.131126 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a6058961bc7450ab489b26158e7dd17) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:44.131543 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c03691d021e4ee19d5bb287ab633a3c with delete_type TABLET_DATA_DELETED (replaced by tablet de0724ba71824a5daaf6f3708b4431b7) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:44.131989 4942 tablet_service.cc:1515] Processing DeleteTablet for tablet 82823fd9ceb1485386616319fa7aeb43 with delete_type TABLET_DATA_DELETED (replaced by tablet f6ec952321924098a142f1438a283aab) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:44.065322 5353 tablet_bootstrap.cc:492] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:44.110671 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69bd106066b6492bb7ea03d15a18df00" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:44.147127 5353 ts_tablet_manager.cc:1397] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.517s user 0.011s sys 0.000s
I20250626 19:57:44.094141 5025 heartbeater.cc:344] Connected to a master server at 127.4.46.189:45039
I20250626 19:57:44.132419 4951 tablet_service.cc:1515] Processing DeleteTablet for tablet 94cefa59de044a36a2afaaf5531128d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:44.151578 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:44.154917 4397 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 147) took 2957 ms. Trace:
I20250626 19:57:44.155211 4397 rpcz_store.cc:276] 0626 19:57:41.197080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.069285 (+2872205us) service_pool.cc:225] Handling call
0626 19:57:44.073289 (+ 4004us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:44.077011 (+ 3722us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:44.154296 (+ 77285us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:44.154578 (+ 282us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:44.154850 (+ 272us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:44.125763 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:44.126282 (+ 519us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:44.126314 (+ 32us) write_op.cc:435] Acquired schema lock
0626 19:57:44.126334 (+ 20us) tablet.cc:660] Decoding operations
0626 19:57:44.126692 (+ 358us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:44.126768 (+ 76us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:44.126792 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:44.126995 (+ 203us) tablet.cc:699] Row locks acquired
0626 19:57:44.127012 (+ 17us) write_op.cc:260] PREPARE: finished
0626 19:57:44.127148 (+ 136us) write_op.cc:270] Start()
0626 19:57:44.127266 (+ 118us) write_op.cc:276] Timestamp: P: 1750967864127160 usec, L: 0
0626 19:57:44.127294 (+ 28us) op_driver.cc:348] REPLICATION: starting
0626 19:57:44.128936 (+ 1642us) log.cc:844] Serialized 433 byte log entry
0626 19:57:44.146265 (+ 17329us) op_driver.cc:464] REPLICATION: finished
0626 19:57:44.151556 (+ 5291us) write_op.cc:301] APPLY: starting
0626 19:57:44.151603 (+ 47us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:44.151693 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:44.151793 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:44.152732 (+ 939us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:44.152888 (+ 156us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:44.152930 (+ 42us) write_op.cc:312] APPLY: finished
0626 19:57:44.153084 (+ 154us) log.cc:844] Serialized 29 byte log entry
0626 19:57:44.153209 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.153267 (+ 58us) write_op.cc:454] Released schema lock
0626 19:57:44.153302 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5165,"mutex_wait_us":47276,"num_ops":2,"prepare.queue_time_us":47439,"prepare.run_cpu_time_us":2442,"prepare.run_wall_time_us":6542,"raft.queue_time_us":1065,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":10,"replication_time_us":18889}]]}
I20250626 19:57:44.159008 4397 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_32a604ea7e0d4d9a8b477c5e6c1e87d5" } new_table_name: "default.Table_9b87a72676524807b2c26f78807b3d9d" modify_external_catalogs: true
I20250626 19:57:44.160017 5965 tablet_replica.cc:331] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:44.150640 5353 raft_consensus.cc:357] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.146483 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "781f8ee8c8ad49c6b2391fcdf20bdfdb" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:44.167759 5353 raft_consensus.cc:383] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:44.180486 5353 raft_consensus.cc:738] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:44.181274 5353 consensus_queue.cc:260] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.182302 4977 raft_consensus.cc:3058] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:44.097581 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6740fcfd3d41487b8a4cb2a6af1e70d4" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:44.190696 4977 raft_consensus.cc:2466] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:44.191077 4668 raft_consensus.cc:3058] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:44.098553 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "609d36d46bb84017b2a3a7ad48bfccfc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:44.161661 5966 tablet.cc:1722] T 2fda659c30c44c8ba402ac5c7f3598e0 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:44.170879 5965 raft_consensus.cc:2241] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:44.098819 5957 consensus_queue.cc:1035] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:44.116611 5887 leader_election.cc:290] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:44.116626 4609 leader_election.cc:304] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
W20250626 19:57:44.200065 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 83570.5us
I20250626 19:57:44.201130 5689 raft_consensus.cc:2802] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:44.201450 5689 raft_consensus.cc:491] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:44.201783 5689 raft_consensus.cc:3058] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:44.175200 4952 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b8473259485460783f62f8193d5b02c with delete_type TABLET_DATA_DELETED (replaced by tablet 8282e22bfb8b4d4994fa8b95ef154a68) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:44.176694 4942 tablet_service.cc:1515] Processing DeleteTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:44.177268 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b66cda4f8ed410184b434d5083deac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7851e464fcd44ecb82404d6a991db2a2) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:44.176088 4951 tablet_service.cc:1515] Processing DeleteTablet for tablet 36997dab0ab3430bba63a90805759b30 with delete_type TABLET_DATA_DELETED (replaced by tablet fafbf98e7f4544b6ad96c1c5ab6c82bc) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:44.198511 4668 raft_consensus.cc:2466] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:44.208632 5689 raft_consensus.cc:513] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.209757 5366 tablet_bootstrap.cc:492] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:44.210167 5366 ts_tablet_manager.cc:1397] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.239s user 0.008s sys 0.001s
I20250626 19:57:44.216111 5689 leader_election.cc:290] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:44.210245 5410 raft_consensus.cc:383] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:44.216707 5410 raft_consensus.cc:738] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:44.217324 5410 consensus_queue.cc:260] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:44.219053 5907 log.cc:927] Time spent T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20250626 19:57:44.220391 5410 ts_tablet_manager.cc:1428] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.127s user 0.005s sys 0.000s
I20250626 19:57:44.224748 5410 tablet_bootstrap.cc:492] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:44.231642 4394 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 4c0338c05a974e0f96865398722c5e8f
I20250626 19:57:44.230008 5366 raft_consensus.cc:357] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.232717 5366 raft_consensus.cc:383] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:44.232893 4394 catalog_manager.cc:6652] ReplaceTablet: tablet 4c0338c05a974e0f96865398722c5e8f deleted and replaced by tablet 34b2919e5f534f4295336a4384be1f64
I20250626 19:57:44.232996 5366 raft_consensus.cc:738] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:44.233228 4394 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34796 (request call id 145) took 3373 ms. Trace:
I20250626 19:57:44.234792 5944 raft_consensus.cc:491] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:44.235183 5944 raft_consensus.cc:513] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.236950 5944 leader_election.cc:290] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:44.237533 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c0338c05a974e0f96865398722c5e8f with delete_type TABLET_DATA_DELETED (replaced by tablet 34b2919e5f534f4295336a4384be1f64) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:44.233495 4394 rpcz_store.cc:276] 0626 19:57:40.859511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:43.971470 (+3111959us) service_pool.cc:225] Handling call
0626 19:57:44.054690 (+ 83220us) spinlock_profiling.cc:243] Waited 81.6 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe791559891 00007fe78f2943a7 00007fe78f294332 00007fe78f294261 00007fe78f2941dc 00007fe78f293492 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:44.233161 (+178471us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:44.147628 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:44.148080 (+ 452us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:44.148108 (+ 28us) write_op.cc:435] Acquired schema lock
0626 19:57:44.148124 (+ 16us) tablet.cc:660] Decoding operations
0626 19:57:44.148456 (+ 332us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:44.148492 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:44.148519 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:44.148753 (+ 234us) tablet.cc:699] Row locks acquired
0626 19:57:44.148773 (+ 20us) write_op.cc:260] PREPARE: finished
0626 19:57:44.148902 (+ 129us) write_op.cc:270] Start()
0626 19:57:44.149007 (+ 105us) write_op.cc:276] Timestamp: P: 1750967864148931 usec, L: 0
0626 19:57:44.149034 (+ 27us) op_driver.cc:348] REPLICATION: starting
0626 19:57:44.149676 (+ 642us) log.cc:844] Serialized 659 byte log entry
0626 19:57:44.172461 (+ 22785us) op_driver.cc:464] REPLICATION: finished
0626 19:57:44.199110 (+ 26649us) write_op.cc:301] APPLY: starting
0626 19:57:44.199176 (+ 66us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:44.199282 (+ 106us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:44.199368 (+ 86us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:44.199863 (+ 495us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:44.200064 (+ 201us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:44.200105 (+ 41us) write_op.cc:312] APPLY: finished
0626 19:57:44.200272 (+ 167us) log.cc:844] Serialized 29 byte log entry
0626 19:57:44.200888 (+ 616us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.200949 (+ 61us) write_op.cc:454] Released schema lock
0626 19:57:44.231062 (+ 30113us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":187599104,"child_traces":[["op",{"apply.queue_time_us":26469,"num_ops":2,"prepare.queue_time_us":8255,"prepare.run_cpu_time_us":2346,"prepare.run_wall_time_us":3488,"replication_time_us":23327}]]}
I20250626 19:57:44.234472 5366 consensus_queue.cc:260] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.247159 5353 ts_tablet_manager.cc:1428] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.100s user 0.009s sys 0.001s
I20250626 19:57:44.248045 5353 tablet_bootstrap.cc:492] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:44.249369 5366 ts_tablet_manager.cc:1428] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.039s user 0.002s sys 0.005s
I20250626 19:57:44.250234 5366 tablet_bootstrap.cc:492] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:44.251017 5798 raft_consensus.cc:2802] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:44.251458 5798 raft_consensus.cc:695] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:44.252223 5798 consensus_queue.cc:237] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.255007 5965 raft_consensus.cc:2270] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:44.257324 5965 ts_tablet_manager.cc:1905] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:44.262347 4951 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c0338c05a974e0f96865398722c5e8f with delete_type TABLET_DATA_DELETED (replaced by tablet 34b2919e5f534f4295336a4384be1f64) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:44.269487 5353 tablet_bootstrap.cc:654] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:44.085806 4399 rpcz_store.cc:269] 0626 19:57:39.852723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:39.852827 (+ 104us) service_pool.cc:225] Handling call
0626 19:57:41.701101 (+1848274us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:41.701138 (+ 37us) write_op.cc:454] Released schema lock
0626 19:57:41.873243 (+172105us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:41.873273 (+ 30us) write_op.cc:454] Released schema lock
0626 19:57:44.032648 (+2159375us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:41.684036 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:41.684421 (+ 385us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:41.684447 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:41.684462 (+ 15us) tablet.cc:660] Decoding operations
0626 19:57:41.684874 (+ 412us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:41.684929 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:41.684951 (+ 22us) tablet.cc:683] Acquiring locks for 3 operations
0626 19:57:41.685193 (+ 242us) tablet.cc:699] Row locks acquired
0626 19:57:41.685206 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:57:41.685313 (+ 107us) write_op.cc:270] Start()
0626 19:57:41.685391 (+ 78us) write_op.cc:276] Timestamp: P: 1750967861685323 usec, L: 0
0626 19:57:41.685407 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:57:41.685876 (+ 469us) log.cc:844] Serialized 1284 byte log entry
0626 19:57:41.693773 (+ 7897us) op_driver.cc:464] REPLICATION: finished
0626 19:57:41.694223 (+ 450us) write_op.cc:301] APPLY: starting
0626 19:57:41.694296 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:41.694412 (+ 116us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:41.694498 (+ 86us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:41.694847 (+ 349us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:41.694994 (+ 147us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0626 19:57:41.695059 (+ 65us) write_op.cc:312] APPLY: finished
0626 19:57:41.695254 (+ 195us) log.cc:844] Serialized 35 byte log entry
0626 19:57:41.695391 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:41.695462 (+ 71us) write_op.cc:454] Released schema lock
0626 19:57:41.695501 (+ 39us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0626 19:57:41.843505 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:41.843891 (+ 386us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:41.843914 (+ 23us) write_op.cc:435] Acquired schema lock
0626 19:57:41.843926 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:41.844229 (+ 303us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:41.844272 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:41.844302 (+ 30us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:41.844458 (+ 156us) tablet.cc:699] Row locks acquired
0626 19:57:41.844468 (+ 10us) write_op.cc:260] PREPARE: finished
0626 19:57:41.844581 (+ 113us) write_op.cc:270] Start()
0626 19:57:41.844663 (+ 82us) write_op.cc:276] Timestamp: P: 1750967861844641 usec, L: 0
0626 19:57:41.844679 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:57:41.845234 (+ 555us) log.cc:844] Serialized 356 byte log entry
0626 19:57:41.852421 (+ 7187us) op_driver.cc:464] REPLICATION: finished
0626 19:57:41.852792 (+ 371us) write_op.cc:301] APPLY: starting
0626 19:57:41.852844 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:41.852930 (+ 86us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:41.853028 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:41.853179 (+ 151us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:41.853294 (+ 115us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:41.853337 (+ 43us) write_op.cc:312] APPLY: finished
0626 19:57:41.853492 (+ 155us) log.cc:844] Serialized 23 byte log entry
0626 19:57:41.853618 (+ 126us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:41.853678 (+ 60us) write_op.cc:454] Released schema lock
0626 19:57:41.853708 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":13456,"reported_tablets":53,"child_traces":[["op",{"apply.queue_time_us":312,"apply.run_cpu_time_us":1445,"apply.run_wall_time_us":1443,"num_ops":3,"prepare.queue_time_us":10089,"prepare.run_cpu_time_us":2149,"prepare.run_wall_time_us":2149,"raft.queue_time_us":3201,"raft.run_cpu_time_us":657,"raft.run_wall_time_us":657,"replication_time_us":8273}],["op",{"apply.queue_time_us":222,"apply.run_cpu_time_us":1096,"apply.run_wall_time_us":1096,"num_ops":1,"prepare.queue_time_us":2234,"prepare.run_cpu_time_us":2027,"prepare.run_wall_time_us":2027,"raft.queue_time_us":2906,"raft.run_cpu_time_us":666,"raft.run_wall_time_us":666,"replication_time_us":7660}]]}
I20250626 19:57:44.236274 4650 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c0338c05a974e0f96865398722c5e8f with delete_type TABLET_DATA_DELETED (replaced by tablet 34b2919e5f534f4295336a4384be1f64) from {username='slave'} at 127.0.0.1:49550
W20250626 19:57:44.239897 4611 leader_election.cc:343] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:44.274448 4726 heartbeater.cc:344] Connected to a master server at 127.4.46.189:45039
W20250626 19:57:44.274278 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 247138us
I20250626 19:57:44.274648 4611 leader_election.cc:304] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:44.274950 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
W20250626 19:57:44.274928 4911 leader_election.cc:343] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Not found: Tablet not found: 609d36d46bb84017b2a3a7ad48bfccfc
I20250626 19:57:44.245266 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef1abf340cc34e00aa67e4588c61c32d" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:44.278205 4755 leader_election.cc:343] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:44.278899 5887 raft_consensus.cc:2802] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
W20250626 19:57:44.247365 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 46090.6us
I20250626 19:57:44.279373 5887 raft_consensus.cc:695] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:44.280138 5887 consensus_queue.cc:237] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.281868 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "781f8ee8c8ad49c6b2391fcdf20bdfdb" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:44.282366 4817 raft_consensus.cc:3058] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:44.287086 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69bd106066b6492bb7ea03d15a18df00" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:44.287679 4820 raft_consensus.cc:3058] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:44.269872 5937 ts_tablet_manager.cc:1918] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:44.288442 5937 log.cc:1199] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/cb34ffcd4da04e32a1220b3f298ec22d
I20250626 19:57:44.289160 5937 ts_tablet_manager.cc:1939] T cb34ffcd4da04e32a1220b3f298ec22d P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:44.290350 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet cb34ffcd4da04e32a1220b3f298ec22d (table default.Table_ff801b0f32eb4c948c82f6c4b7d42b97 [id=46243e06f82542189b78751aef158149]) successfully deleted
W20250626 19:57:44.295738 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 6b8473259485460783f62f8193d5b02c because the tablet was not found. No further retry: Not found: Tablet not found: 6b8473259485460783f62f8193d5b02c
I20250626 19:57:44.297572 4820 raft_consensus.cc:2466] T 69bd106066b6492bb7ea03d15a18df00 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:44.298779 4609 leader_election.cc:304] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:44.299468 4399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_53d2cf225a36435fbf6c220b550f8bba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:44.300691 5689 raft_consensus.cc:2802] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:44.301198 5937 tablet_replica.cc:331] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:44.301767 4394 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_d8774697646e49e9a77cfb883bae0f98" } modify_external_catalogs: true
I20250626 19:57:44.301929 4817 raft_consensus.cc:2466] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:44.302340 5937 raft_consensus.cc:2241] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:44.302256 4394 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_d8774697646e49e9a77cfb883bae0f98" } modify_external_catalogs: true
I20250626 19:57:44.301143 5689 raft_consensus.cc:695] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:44.303303 5689 consensus_queue.cc:237] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.305068 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef1abf340cc34e00aa67e4588c61c32d" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:44.313210 4391 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet f6ec952321924098a142f1438a283aab
I20250626 19:57:44.313784 5937 raft_consensus.cc:2270] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:44.315735 4650 tablet_service.cc:1515] Processing DeleteTablet for tablet f6ec952321924098a142f1438a283aab with delete_type TABLET_DATA_DELETED (replaced by tablet 2d0a30ee16d043f9837dce612dae670a) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:44.316838 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet f6ec952321924098a142f1438a283aab with delete_type TABLET_DATA_DELETED (replaced by tablet 2d0a30ee16d043f9837dce612dae670a) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:44.317067 4951 tablet_service.cc:1515] Processing DeleteTablet for tablet f6ec952321924098a142f1438a283aab with delete_type TABLET_DATA_DELETED (replaced by tablet 2d0a30ee16d043f9837dce612dae670a) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:44.319034 5937 ts_tablet_manager.cc:1905] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:44.319201 4391 catalog_manager.cc:6652] ReplaceTablet: tablet f6ec952321924098a142f1438a283aab deleted and replaced by tablet 2d0a30ee16d043f9837dce612dae670a
I20250626 19:57:44.319288 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69bd106066b6492bb7ea03d15a18df00" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:44.319633 4391 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34796 (request call id 148) took 2264 ms. Trace:
I20250626 19:57:44.325013 4390 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 144) took 3749 ms. Trace:
I20250626 19:57:44.325320 4390 rpcz_store.cc:276] 0626 19:57:40.575303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:43.965714 (+3390411us) service_pool.cc:225] Handling call
0626 19:57:44.055289 (+ 89575us) spinlock_profiling.cc:243] Waited 87.5 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe791556d36 00007fe78f2736a4 00007fe78f273632 00007fe78f273561 00007fe78f2734dc 00007fe78f272792 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:44.079917 (+ 24628us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:44.104594 (+ 24677us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:44.324420 (+219826us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:44.324693 (+ 273us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:44.324960 (+ 267us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:44.228244 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:44.228696 (+ 452us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:44.228720 (+ 24us) write_op.cc:435] Acquired schema lock
0626 19:57:44.228738 (+ 18us) tablet.cc:660] Decoding operations
0626 19:57:44.229036 (+ 298us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:44.229077 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:44.229108 (+ 31us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:44.229318 (+ 210us) tablet.cc:699] Row locks acquired
0626 19:57:44.229336 (+ 18us) write_op.cc:260] PREPARE: finished
0626 19:57:44.229452 (+ 116us) write_op.cc:270] Start()
0626 19:57:44.229562 (+ 110us) write_op.cc:276] Timestamp: P: 1750967864229513 usec, L: 0
0626 19:57:44.229581 (+ 19us) op_driver.cc:348] REPLICATION: starting
0626 19:57:44.230075 (+ 494us) log.cc:844] Serialized 433 byte log entry
0626 19:57:44.307614 (+ 77539us) op_driver.cc:464] REPLICATION: finished
0626 19:57:44.322511 (+ 14897us) write_op.cc:301] APPLY: starting
0626 19:57:44.322569 (+ 58us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:44.322670 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:44.322789 (+ 119us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:44.323207 (+ 418us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:44.323339 (+ 132us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:44.323384 (+ 45us) write_op.cc:312] APPLY: finished
0626 19:57:44.323567 (+ 183us) log.cc:844] Serialized 29 byte log entry
0626 19:57:44.324152 (+ 585us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.324215 (+ 63us) write_op.cc:454] Released schema lock
0626 19:57:44.324251 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":201224448,"child_traces":[["op",{"apply.queue_time_us":14724,"apply.run_cpu_time_us":1531,"apply.run_wall_time_us":4469,"mutex_wait_us":103858,"num_ops":2,"prepare.queue_time_us":121451,"prepare.run_cpu_time_us":2157,"prepare.run_wall_time_us":65220,"raft.queue_time_us":83,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":222,"replication_time_us":77954,"threads_started":1}]]}
I20250626 19:57:44.320014 4391 rpcz_store.cc:276] 0626 19:57:42.054790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.083786 (+2028996us) service_pool.cc:225] Handling call
0626 19:57:44.319526 (+235740us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:44.178293 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:44.225293 (+ 47000us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:44.225331 (+ 38us) write_op.cc:435] Acquired schema lock
0626 19:57:44.225351 (+ 20us) tablet.cc:660] Decoding operations
0626 19:57:44.225715 (+ 364us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:44.225767 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:44.225798 (+ 31us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:44.226023 (+ 225us) tablet.cc:699] Row locks acquired
0626 19:57:44.226042 (+ 19us) write_op.cc:260] PREPARE: finished
0626 19:57:44.226171 (+ 129us) write_op.cc:270] Start()
0626 19:57:44.226267 (+ 96us) write_op.cc:276] Timestamp: P: 1750967864226232 usec, L: 0
0626 19:57:44.226287 (+ 20us) op_driver.cc:348] REPLICATION: starting
0626 19:57:44.226893 (+ 606us) log.cc:844] Serialized 625 byte log entry
0626 19:57:44.307358 (+ 80465us) op_driver.cc:464] REPLICATION: finished
0626 19:57:44.309459 (+ 2101us) write_op.cc:301] APPLY: starting
0626 19:57:44.309518 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:44.309619 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:44.309705 (+ 86us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:44.310020 (+ 315us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:44.310197 (+ 177us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:44.310241 (+ 44us) write_op.cc:312] APPLY: finished
0626 19:57:44.310390 (+ 149us) log.cc:844] Serialized 29 byte log entry
0626 19:57:44.310523 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.310581 (+ 58us) write_op.cc:454] Released schema lock
0626 19:57:44.310617 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1979,"apply.run_cpu_time_us":1351,"apply.run_wall_time_us":12639,"num_ops":2,"prepare.queue_time_us":3736,"prepare.run_cpu_time_us":2369,"prepare.run_wall_time_us":49800,"replication_time_us":80975}]]}
I20250626 19:57:44.341614 5944 raft_consensus.cc:491] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:44.342082 5944 raft_consensus.cc:513] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.343977 5944 leader_election.cc:290] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
W20250626 19:57:44.345825 4611 leader_election.cc:343] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:44.346045 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c5aba053ca4535ad2376b2066cfe47" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:44.347188 4755 leader_election.cc:343] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:44.347976 4912 leader_election.cc:343] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:44.348384 4912 leader_election.cc:304] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
W20250626 19:57:44.348990 4912 raft_consensus.cc:2684] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250626 19:57:44.349548 4756 leader_election.cc:343] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:44.349921 4756 leader_election.cc:304] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:44.351145 5506 raft_consensus.cc:2747] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:44.352953 4911 consensus_peers.cc:487] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Couldn't send request to peer 93f885c6802341c889a40e9c20fbfff1. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:44.357677 4399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53d2cf225a36435fbf6c220b550f8bba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:44.366839 5410 tablet_bootstrap.cc:654] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:44.371369 5807 consensus_queue.cc:1035] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:44.373315 5921 consensus_queue.cc:1035] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:44.416110 4390 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_54b68b0dd5ba469cba186a5d4c6a9941"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:44.417716 4390 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54b68b0dd5ba469cba186a5d4c6a9941 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:44.423308 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_12a980fb917e4db18bfc5eeacda0d9d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:44.424777 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12a980fb917e4db18bfc5eeacda0d9d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:44.404095 5936 ts_tablet_manager.cc:1918] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:44.439232 5936 log.cc:1199] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/cb34ffcd4da04e32a1220b3f298ec22d
I20250626 19:57:44.441308 5936 ts_tablet_manager.cc:1939] T cb34ffcd4da04e32a1220b3f298ec22d P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:44.456506 5366 tablet_bootstrap.cc:654] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:44.461395 5965 ts_tablet_manager.cc:1918] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:44.461863 5965 log.cc:1199] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/609d36d46bb84017b2a3a7ad48bfccfc
I20250626 19:57:44.464533 5410 tablet_bootstrap.cc:492] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:44.473354 5410 ts_tablet_manager.cc:1397] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.249s user 0.010s sys 0.004s
I20250626 19:57:44.478446 5965 ts_tablet_manager.cc:1939] T 609d36d46bb84017b2a3a7ad48bfccfc P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:44.480230 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 609d36d46bb84017b2a3a7ad48bfccfc (table default.Table_d23ee99af7b845b78d2200b962f681a6 [id=fc725d6df6774d4e922e2d98d461b111]) successfully deleted
W20250626 19:57:44.480859 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet f43aed8f07844c5ca727ae6df63896df because the tablet was not found. No further retry: Not found: Tablet not found: f43aed8f07844c5ca727ae6df63896df
W20250626 19:57:44.481356 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 94327cab01df4338aaec157736d106f6 because the tablet was not found. No further retry: Not found: Tablet not found: 94327cab01df4338aaec157736d106f6
W20250626 19:57:44.481870 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 because the tablet was not found. No further retry: Not found: Tablet not found: dd0ca2fab40f45e5a1f9bd6059ca1f08
W20250626 19:57:44.482383 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 3c03691d021e4ee19d5bb287ab633a3c because the tablet was not found. No further retry: Not found: Tablet not found: 3c03691d021e4ee19d5bb287ab633a3c
I20250626 19:57:44.482496 5965 tablet_replica.cc:331] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
W20250626 19:57:44.462845 4912 consensus_peers.cc:487] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb34ffcd4da04e32a1220b3f298ec22d. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:44.496351 5937 ts_tablet_manager.cc:1918] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:44.499125 5944 raft_consensus.cc:491] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 19:57:44.499861 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 82823fd9ceb1485386616319fa7aeb43 because the tablet was not found. No further retry: Not found: Tablet not found: 82823fd9ceb1485386616319fa7aeb43
I20250626 19:57:44.499748 5944 raft_consensus.cc:513] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:44.500569 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 94cefa59de044a36a2afaaf5531128d8 because the tablet was not found. No further retry: Not found: Tablet not found: 94cefa59de044a36a2afaaf5531128d8
W20250626 19:57:44.501133 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 6b8473259485460783f62f8193d5b02c because the tablet was not found. No further retry: Not found: Tablet not found: 6b8473259485460783f62f8193d5b02c
I20250626 19:57:44.502185 5944 leader_election.cc:290] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:44.506771 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83f0431638144710b598b3f87d9a576e" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:44.507191 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c5aba053ca4535ad2376b2066cfe47" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:44.507740 5410 raft_consensus.cc:357] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:44.508347 5410 raft_consensus.cc:383] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:44.508579 5410 raft_consensus.cc:738] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:44.509099 5410 consensus_queue.cc:260] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
W20250626 19:57:44.509588 4756 leader_election.cc:343] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:44.510008 4756 leader_election.cc:304] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:44.510910 5944 raft_consensus.cc:2747] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:44.512324 4755 leader_election.cc:343] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:44.515120 5410 ts_tablet_manager.cc:1428] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20250626 19:57:44.516552 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83f0431638144710b598b3f87d9a576e" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:44.528470 5937 log.cc:1199] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/36997dab0ab3430bba63a90805759b30
I20250626 19:57:44.529198 5937 ts_tablet_manager.cc:1939] T 36997dab0ab3430bba63a90805759b30 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:44.534719 4392 catalog_manager.cc:5582] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:44.536026 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:44.539592 4392 catalog_manager.cc:5582] T 211f6137d9b84afa9aee6432440f7eb8 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } }
W20250626 19:57:44.542199 4756 leader_election.cc:343] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:44.542647 4756 leader_election.cc:304] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:44.543605 5944 raft_consensus.cc:2747] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:44.543121 4392 catalog_manager.cc:5582] T e8122b26848141a39d2462dd3491f9cd P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:44.549191 4390 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_b7a7e36d615144f692ba3b32abc5fb5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:44.551008 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 2fda659c30c44c8ba402ac5c7f3598e0 (table default.Table_4708ce225be445acbb30bbcc243949bc [id=ae81e1731bc1461385bf84e443ff0730]). Expected version 1 got 0
W20250626 19:57:44.551952 4390 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7a7e36d615144f692ba3b32abc5fb5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:44.552721 4392 catalog_manager.cc:5582] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } }
I20250626 19:57:44.561215 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:44.566002 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:44.566598 4392 catalog_manager.cc:5582] T 5706841a14bd4ece965098b88ccb7ec8 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:44.569988 4392 catalog_manager.cc:5582] T 1ba78c1b40ef4925b4108e1455bcfc44 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:44.572686 4392 catalog_manager.cc:5582] T e65b16141c3d47c5b66024da59b9bbe9 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:44.574837 4392 catalog_manager.cc:5582] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } }
I20250626 19:57:44.576423 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:44.581719 4392 catalog_manager.cc:5582] T a3636b78f4a64f9eacb4e5979768929b P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:44.584729 5410 tablet_bootstrap.cc:492] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:44.587604 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:44.588178 4392 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (table default.Table_8b997467fa764e8fbf273c3b1f2ccb3e [id=760c56838ee140d78aa368a3810d19a8]). Expected version 1 got 0
I20250626 19:57:44.589761 5965 raft_consensus.cc:2241] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:44.590672 5965 pending_rounds.cc:70] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6: Trying to abort 1 pending ops.
I20250626 19:57:44.591051 5965 pending_rounds.cc:77] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964367455731712 op_type: NO_OP noop_request { }
I20250626 19:57:44.591789 5965 raft_consensus.cc:2887] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:57:44.592320 5965 raft_consensus.cc:2270] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:44.595487 5965 ts_tablet_manager.cc:1905] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:44.604951 5353 tablet_bootstrap.cc:492] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:44.605865 5353 ts_tablet_manager.cc:1397] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.358s user 0.037s sys 0.051s
I20250626 19:57:44.611419 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet cb34ffcd4da04e32a1220b3f298ec22d (table default.Table_ff801b0f32eb4c948c82f6c4b7d42b97 [id=46243e06f82542189b78751aef158149]) successfully deleted
W20250626 19:57:44.614538 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 609d36d46bb84017b2a3a7ad48bfccfc because the tablet was not found. No further retry: Not found: Tablet not found: 609d36d46bb84017b2a3a7ad48bfccfc
I20250626 19:57:44.615546 5936 tablet_replica.cc:331] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
W20250626 19:57:44.623106 4462 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.46.190:33823 timed out after 0.642s (SENT) [suppressed 1 similar messages]
I20250626 19:57:44.634495 5410 tablet_bootstrap.cc:654] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:44.641832 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 36997dab0ab3430bba63a90805759b30 (table default.Table_c010e84626544214a45a911f02c4d903 [id=1b4629c56e5c4bf3b69fabbfb10b8008]) successfully deleted
W20250626 19:57:44.642542 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 94cefa59de044a36a2afaaf5531128d8 because the tablet was not found. No further retry: Not found: Tablet not found: 94cefa59de044a36a2afaaf5531128d8
W20250626 19:57:44.644657 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 6b66cda4f8ed410184b434d5083deac4 because the tablet was not found. No further retry: Not found: Tablet not found: 6b66cda4f8ed410184b434d5083deac4
W20250626 19:57:44.646366 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet dd0ca2fab40f45e5a1f9bd6059ca1f08 because the tablet was not found. No further retry: Not found: Tablet not found: dd0ca2fab40f45e5a1f9bd6059ca1f08
W20250626 19:57:44.648161 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet fc526a05990446a3b3bc973e79511476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc526a05990446a3b3bc973e79511476 already in progress: creating tablet
W20250626 19:57:44.648823 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet f807cf54bbb146ff989762c18410e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f807cf54bbb146ff989762c18410e6bb already in progress: creating tablet
W20250626 19:57:44.650569 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet f43aed8f07844c5ca727ae6df63896df because the tablet was not found. No further retry: Not found: Tablet not found: f43aed8f07844c5ca727ae6df63896df
I20250626 19:57:44.651437 5353 raft_consensus.cc:357] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.652266 5353 raft_consensus.cc:383] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:44.652880 5353 raft_consensus.cc:738] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:44.654090 5353 consensus_queue.cc:260] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.657701 5353 ts_tablet_manager.cc:1428] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.051s user 0.005s sys 0.000s
I20250626 19:57:44.661998 5353 tablet_bootstrap.cc:492] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:44.677973 5936 raft_consensus.cc:2241] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:44.678580 5936 raft_consensus.cc:2270] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:44.680755 5936 ts_tablet_manager.cc:1905] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:44.699740 5366 tablet_bootstrap.cc:492] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:44.699983 5353 tablet_bootstrap.cc:654] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:44.700570 5366 ts_tablet_manager.cc:1397] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.450s user 0.011s sys 0.003s
I20250626 19:57:44.712531 4396 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 7851e464fcd44ecb82404d6a991db2a2 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]). Expected version 1 got 0
I20250626 19:57:44.713889 5937 tablet_replica.cc:331] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:44.715144 5937 raft_consensus.cc:2241] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:44.716100 5937 raft_consensus.cc:2270] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:44.715387 4396 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:44.717254 4396 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 2fda659c30c44c8ba402ac5c7f3598e0 (table default.Table_4708ce225be445acbb30bbcc243949bc [id=ae81e1731bc1461385bf84e443ff0730]). Expected version 1 got 0
I20250626 19:57:44.720949 4396 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:44.724154 5936 ts_tablet_manager.cc:1918] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:57:44.724571 5936 log.cc:1199] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/4c0338c05a974e0f96865398722c5e8f
I20250626 19:57:44.725080 4396 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:44.725330 5936 ts_tablet_manager.cc:1939] T 4c0338c05a974e0f96865398722c5e8f P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:44.728011 4396 catalog_manager.cc:5582] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } }
W20250626 19:57:44.729512 5907 log.cc:927] Time spent T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Append to log took a long time: real 0.117s user 0.002s sys 0.000s
I20250626 19:57:44.730448 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_1fc5753cbe224eb4b476617a5d5d0d31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:44.733374 5794 consensus_queue.cc:1035] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250626 19:57:44.731961 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fc5753cbe224eb4b476617a5d5d0d31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:44.730728 4396 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:44.753225 4396 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:44.756072 4396 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (table default.Table_8b997467fa764e8fbf273c3b1f2ccb3e [id=760c56838ee140d78aa368a3810d19a8]). Expected version 1 got 0
I20250626 19:57:44.773259 5410 tablet_bootstrap.cc:492] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:44.773725 5410 ts_tablet_manager.cc:1397] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.189s user 0.030s sys 0.013s
I20250626 19:57:44.777374 5410 raft_consensus.cc:357] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.778014 5410 raft_consensus.cc:383] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:44.777755 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_3988af86d47e4eaa9103970949e16f12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:44.778251 5410 raft_consensus.cc:738] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
W20250626 19:57:44.779222 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3988af86d47e4eaa9103970949e16f12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:44.800297 5937 ts_tablet_manager.cc:1905] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:44.803730 5366 raft_consensus.cc:357] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.804339 5366 raft_consensus.cc:383] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:44.804581 5366 raft_consensus.cc:738] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:44.805832 5366 consensus_queue.cc:260] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.818706 5949 raft_consensus.cc:491] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:44.819235 5949 raft_consensus.cc:513] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.819751 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_ec5124d21b3d42e294ed85b470fbe5e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:44.821193 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec5124d21b3d42e294ed85b470fbe5e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:44.821230 5949 leader_election.cc:290] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:44.822039 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3143181fae16411888d74f9152f3d5cc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:44.822650 4668 raft_consensus.cc:2466] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:44.822914 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3143181fae16411888d74f9152f3d5cc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:44.823516 4817 raft_consensus.cc:2466] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:44.823921 4912 leader_election.cc:304] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:44.827041 5410 consensus_queue.cc:260] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.829129 5410 ts_tablet_manager.cc:1428] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.055s user 0.004s sys 0.002s
I20250626 19:57:44.829952 5410 tablet_bootstrap.cc:492] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:44.832787 5949 raft_consensus.cc:2802] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:44.833186 5949 raft_consensus.cc:491] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:44.833570 5949 raft_consensus.cc:3058] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:44.865957 5366 ts_tablet_manager.cc:1428] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.164s user 0.008s sys 0.000s
I20250626 19:57:44.869095 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_1bc851fd108147e3ab051386bab41342"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:44.870602 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bc851fd108147e3ab051386bab41342 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:44.883600 5366 tablet_bootstrap.cc:492] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:44.891940 4391 master_service.cc:948] ReplaceTablet: received request to replace tablet 65af5ae255d74143976392bccc50f3ba from {username='slave'} at 127.0.0.1:34796
W20250626 19:57:44.892022 4386 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (table default.Table_8b997467fa764e8fbf273c3b1f2ccb3e [id=760c56838ee140d78aa368a3810d19a8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 consensus state has no leader
I20250626 19:57:44.907083 5949 raft_consensus.cc:513] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.909780 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3143181fae16411888d74f9152f3d5cc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:44.910370 4671 raft_consensus.cc:3058] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:44.915707 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]) successfully deleted
I20250626 19:57:44.918800 5353 tablet_bootstrap.cc:492] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:44.919490 5353 ts_tablet_manager.cc:1397] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.258s user 0.007s sys 0.011s
I20250626 19:57:44.922041 5353 raft_consensus.cc:357] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:44.922712 5353 raft_consensus.cc:383] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:44.922978 5353 raft_consensus.cc:738] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:44.923956 5936 tablet_replica.cc:331] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:44.924816 5936 raft_consensus.cc:2241] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:44.925338 5936 raft_consensus.cc:2270] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:44.942588 4391 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 65af5ae255d74143976392bccc50f3ba
I20250626 19:57:44.943944 4391 catalog_manager.cc:6652] ReplaceTablet: tablet 65af5ae255d74143976392bccc50f3ba deleted and replaced by tablet a8ef6f024d0642d4a6dd67d646787868
I20250626 19:57:44.945860 4951 tablet_service.cc:1515] Processing DeleteTablet for tablet 65af5ae255d74143976392bccc50f3ba with delete_type TABLET_DATA_DELETED (replaced by tablet a8ef6f024d0642d4a6dd67d646787868) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:44.946050 4650 tablet_service.cc:1515] Processing DeleteTablet for tablet 65af5ae255d74143976392bccc50f3ba with delete_type TABLET_DATA_DELETED (replaced by tablet a8ef6f024d0642d4a6dd67d646787868) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:44.946707 5410 tablet_bootstrap.cc:654] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:44.947175 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 65af5ae255d74143976392bccc50f3ba with delete_type TABLET_DATA_DELETED (replaced by tablet a8ef6f024d0642d4a6dd67d646787868) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:44.948787 5366 tablet_bootstrap.cc:654] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:44.961083 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 8f54a2d0098446a5b60dc6d07a234bc4 (DEFAULT_TABLE table=default.Table_6e9e80ead85b44f19cd580a82983fe06 [id=e219f91991754e449ef8b14aaad287e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:44.962168 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f54a2d0098446a5b60dc6d07a234bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:44.968911 4650 tablet_service.cc:1468] Processing CreateTablet for tablet 8f54a2d0098446a5b60dc6d07a234bc4 (DEFAULT_TABLE table=default.Table_6e9e80ead85b44f19cd580a82983fe06 [id=e219f91991754e449ef8b14aaad287e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:44.968850 4951 tablet_service.cc:1468] Processing CreateTablet for tablet 8f54a2d0098446a5b60dc6d07a234bc4 (DEFAULT_TABLE table=default.Table_6e9e80ead85b44f19cd580a82983fe06 [id=e219f91991754e449ef8b14aaad287e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:44.970311 4957 tablet_service.cc:1468] Processing CreateTablet for tablet 56f319c748374b7cbbf037ef55aecd1b (DEFAULT_TABLE table=default.Table_bf54a23958c14a1699f712665f29c0ad [id=a1cfdad9d9584a04be9e39bb35c60fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:44.974066 4391 master_service.cc:948] ReplaceTablet: received request to replace tablet e94a73b50b564a97bbee0f5015bb8805 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:44.974548 4649 tablet_service.cc:1468] Processing CreateTablet for tablet 56f319c748374b7cbbf037ef55aecd1b (DEFAULT_TABLE table=default.Table_bf54a23958c14a1699f712665f29c0ad [id=a1cfdad9d9584a04be9e39bb35c60fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:44.981623 4951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f54a2d0098446a5b60dc6d07a234bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:44.983330 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56f319c748374b7cbbf037ef55aecd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:44.983455 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 56f319c748374b7cbbf037ef55aecd1b (DEFAULT_TABLE table=default.Table_bf54a23958c14a1699f712665f29c0ad [id=a1cfdad9d9584a04be9e39bb35c60fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:44.984746 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56f319c748374b7cbbf037ef55aecd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:44.989456 4942 tablet_service.cc:1468] Processing CreateTablet for tablet 11648a3a9a94450284b8bc14ffe95852 (DEFAULT_TABLE table=default.Table_c46f810d71934cdeb268c25bc98d2325 [id=cbc1cad5911e4ed0b873ea91e1824140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:44.990211 4650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f54a2d0098446a5b60dc6d07a234bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:44.990513 4942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11648a3a9a94450284b8bc14ffe95852. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:44.991314 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56f319c748374b7cbbf037ef55aecd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:44.995487 5887 raft_consensus.cc:491] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:44.995937 5887 raft_consensus.cc:513] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:44.997947 5887 leader_election.cc:290] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
W20250626 19:57:45.001164 4611 leader_election.cc:343] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Not found: Tablet not found: 609d36d46bb84017b2a3a7ad48bfccfc
I20250626 19:57:45.004770 5965 ts_tablet_manager.cc:1918] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:45.006083 5965 log.cc:1199] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/cb34ffcd4da04e32a1220b3f298ec22d
W20250626 19:57:45.016664 4288 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 48568.6us
I20250626 19:57:45.017216 4652 tablet_service.cc:1468] Processing CreateTablet for tablet 11648a3a9a94450284b8bc14ffe95852 (DEFAULT_TABLE table=default.Table_c46f810d71934cdeb268c25bc98d2325 [id=cbc1cad5911e4ed0b873ea91e1824140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:45.018359 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11648a3a9a94450284b8bc14ffe95852. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:45.019402 4609 leader_election.cc:343] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Not found: Tablet not found: 609d36d46bb84017b2a3a7ad48bfccfc
I20250626 19:57:45.019765 4609 leader_election.cc:304] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:45.020709 4391 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet e94a73b50b564a97bbee0f5015bb8805
I20250626 19:57:45.020684 5887 raft_consensus.cc:2747] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:45.023249 4391 catalog_manager.cc:6652] ReplaceTablet: tablet e94a73b50b564a97bbee0f5015bb8805 deleted and replaced by tablet 9e4673214ad84f75a861c3af82a13d8f
I20250626 19:57:45.023206 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3143181fae16411888d74f9152f3d5cc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:45.023785 4816 raft_consensus.cc:3058] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:45.024042 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_b574eeb305114df283c62fab66bdafb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:45.026028 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b574eeb305114df283c62fab66bdafb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:45.028306 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet e94a73b50b564a97bbee0f5015bb8805 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e4673214ad84f75a861c3af82a13d8f) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:45.028319 4952 tablet_service.cc:1515] Processing DeleteTablet for tablet e94a73b50b564a97bbee0f5015bb8805 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e4673214ad84f75a861c3af82a13d8f) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:45.029871 4793 tablet_service.cc:1515] Processing DeleteTablet for tablet e94a73b50b564a97bbee0f5015bb8805 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e4673214ad84f75a861c3af82a13d8f) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:45.008564 5965 ts_tablet_manager.cc:1939] T cb34ffcd4da04e32a1220b3f298ec22d P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:45.017803 5949 leader_election.cc:290] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:45.037415 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:45.010535 4671 raft_consensus.cc:2466] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:45.049471 4793 tablet_service.cc:1468] Processing CreateTablet for tablet 11648a3a9a94450284b8bc14ffe95852 (DEFAULT_TABLE table=default.Table_c46f810d71934cdeb268c25bc98d2325 [id=cbc1cad5911e4ed0b873ea91e1824140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:45.049916 4912 leader_election.cc:304] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:45.050716 4793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11648a3a9a94450284b8bc14ffe95852. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:45.051254 5924 raft_consensus.cc:2802] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:45.051690 5924 raft_consensus.cc:695] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:45.052629 5924 consensus_queue.cc:237] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:45.056097 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet cb34ffcd4da04e32a1220b3f298ec22d (table default.Table_ff801b0f32eb4c948c82f6c4b7d42b97 [id=46243e06f82542189b78751aef158149]) successfully deleted
W20250626 19:57:45.060930 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 6b66cda4f8ed410184b434d5083deac4 because the tablet was not found. No further retry: Not found: Tablet not found: 6b66cda4f8ed410184b434d5083deac4
I20250626 19:57:45.061131 5965 tablet_replica.cc:331] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
W20250626 19:57:45.062031 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 36997dab0ab3430bba63a90805759b30 because the tablet was not found. No further retry: Not found: Tablet not found: 36997dab0ab3430bba63a90805759b30
I20250626 19:57:45.062031 5965 raft_consensus.cc:2241] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:45.062803 4794 tablet_service.cc:1468] Processing CreateTablet for tablet 2d0a30ee16d043f9837dce612dae670a (DEFAULT_TABLE table=default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:45.064356 4794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d0a30ee16d043f9837dce612dae670a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:45.073361 4952 tablet_service.cc:1468] Processing CreateTablet for tablet 2d0a30ee16d043f9837dce612dae670a (DEFAULT_TABLE table=default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:45.075189 4952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d0a30ee16d043f9837dce612dae670a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:45.085450 5936 ts_tablet_manager.cc:1905] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:45.093663 5353 consensus_queue.cc:260] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:45.099345 5366 tablet_bootstrap.cc:492] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:45.097930 5937 ts_tablet_manager.cc:1918] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:57:45.099776 5366 ts_tablet_manager.cc:1397] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.220s user 0.010s sys 0.009s
I20250626 19:57:45.101080 5937 log.cc:1199] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/4c0338c05a974e0f96865398722c5e8f
I20250626 19:57:45.103610 5366 raft_consensus.cc:357] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:45.104521 5366 raft_consensus.cc:383] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:45.104796 5366 raft_consensus.cc:738] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
W20250626 19:57:45.107213 4611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 47638.9us
I20250626 19:57:45.109026 4657 tablet_service.cc:1468] Processing CreateTablet for tablet 2d0a30ee16d043f9837dce612dae670a (DEFAULT_TABLE table=default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:45.110383 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d0a30ee16d043f9837dce612dae670a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:45.122563 5965 raft_consensus.cc:2270] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:45.108896 5937 ts_tablet_manager.cc:1939] T 4c0338c05a974e0f96865398722c5e8f P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:45.034612 4816 raft_consensus.cc:2466] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:45.137569 5965 ts_tablet_manager.cc:1905] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:45.139834 5794 consensus_queue.cc:1035] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:45.154645 5937 tablet_replica.cc:331] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:45.155603 5937 raft_consensus.cc:2241] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:45.156203 5937 raft_consensus.cc:2270] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:45.165475 5353 ts_tablet_manager.cc:1428] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.246s user 0.006s sys 0.000s
I20250626 19:57:45.166481 5353 tablet_bootstrap.cc:492] T ef1abf340cc34e00aa67e4588c61c32d P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:45.170334 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]) successfully deleted
I20250626 19:57:45.173286 5410 tablet_bootstrap.cc:492] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:45.173735 5410 ts_tablet_manager.cc:1397] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.344s user 0.009s sys 0.000s
I20250626 19:57:45.182457 5366 consensus_queue.cc:260] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:45.196529 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (DEFAULT_TABLE table=default.Table_e6347deb7eb14f08bd5e02821a5a55dd [id=e3400f897e9944549162854e3f92ac21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:45.196193 5937 ts_tablet_manager.cc:1905] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:45.223184 5366 ts_tablet_manager.cc:1428] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.123s user 0.005s sys 0.002s
I20250626 19:57:45.223417 4795 tablet_service.cc:1468] Processing CreateTablet for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (DEFAULT_TABLE table=default.Table_e6347deb7eb14f08bd5e02821a5a55dd [id=e3400f897e9944549162854e3f92ac21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:45.212631 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7def0bb1dbdc494eaa71748ebe1f8847. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:45.198961 5410 raft_consensus.cc:357] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:45.232997 5410 raft_consensus.cc:383] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:45.234014 4654 tablet_service.cc:1468] Processing CreateTablet for tablet 34b2919e5f534f4295336a4384be1f64 (DEFAULT_TABLE table=default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:45.235419 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b2919e5f534f4295336a4384be1f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:45.242854 5410 raft_consensus.cc:738] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:45.245805 5410 consensus_queue.cc:260] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:45.251170 5907 log.cc:927] Time spent T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Append to log took a long time: real 0.056s user 0.003s sys 0.000s
I20250626 19:57:45.260172 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 34b2919e5f534f4295336a4384be1f64 (DEFAULT_TABLE table=default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:45.272647 5410 ts_tablet_manager.cc:1428] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.099s user 0.008s sys 0.000s
I20250626 19:57:45.283046 4954 tablet_service.cc:1468] Processing CreateTablet for tablet 34b2919e5f534f4295336a4384be1f64 (DEFAULT_TABLE table=default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:45.284175 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b2919e5f534f4295336a4384be1f64. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:45.287199 4462 leader_election.cc:336] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823): Timed out: RequestConsensusVote RPC to 127.4.46.190:33823 timed out after 0.642s (SENT)
I20250626 19:57:45.287756 4462 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31; no voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:45.296545 4952 tablet_service.cc:1468] Processing CreateTablet for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (DEFAULT_TABLE table=default.Table_e6347deb7eb14f08bd5e02821a5a55dd [id=e3400f897e9944549162854e3f92ac21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:45.297973 4952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7def0bb1dbdc494eaa71748ebe1f8847. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:45.302817 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_3f76c8e8c67b452cb9dd1c2817f493b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:45.304509 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f76c8e8c67b452cb9dd1c2817f493b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:45.316114 5353 tablet_bootstrap.cc:654] T ef1abf340cc34e00aa67e4588c61c32d P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:45.321790 5999 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 19:57:45.335062 5366 tablet_bootstrap.cc:492] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:45.344998 5965 ts_tablet_manager.cc:1918] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:57:45.345418 5965 log.cc:1199] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/4c0338c05a974e0f96865398722c5e8f
I20250626 19:57:45.346140 5965 ts_tablet_manager.cc:1939] T 4c0338c05a974e0f96865398722c5e8f P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:45.370203 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:45.373031 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 7851e464fcd44ecb82404d6a991db2a2 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]). Expected version 1 got 0
I20250626 19:57:45.376622 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7def0bb1dbdc494eaa71748ebe1f8847. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:45.377476 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b2919e5f534f4295336a4384be1f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:45.394258 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:45.393986 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_ec900845fd1c45e68e99858c077c57ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:45.394858 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
W20250626 19:57:45.395505 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec900845fd1c45e68e99858c077c57ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:45.397385 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:45.399118 5965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 171) took 1137 ms. Trace:
I20250626 19:57:45.399477 5965 rpcz_store.cc:276] 0626 19:57:44.262038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.262192 (+ 154us) service_pool.cc:225] Handling call
0626 19:57:45.060966 (+798774us) ts_tablet_manager.cc:1145] Deleting tablet 4c0338c05a974e0f96865398722c5e8f
0626 19:57:45.061005 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:45.275157 (+214152us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:45.344757 (+ 69600us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:45.399043 (+ 54286us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8546,"tablet-delete.queue_time_us":798149}
I20250626 19:57:45.400905 5965 tablet_replica.cc:331] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:45.401757 5965 raft_consensus.cc:2241] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:45.403095 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 4c0338c05a974e0f96865398722c5e8f (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]) successfully deleted
I20250626 19:57:45.402652 5965 raft_consensus.cc:2270] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:45.413199 4399 catalog_manager.cc:5582] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:45.331971 5996 consensus_queue.cc:1035] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:45.349642 5991 raft_consensus.cc:491] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:45.358871 4727 heartbeater.cc:344] Connected to a master server at 127.4.46.190:33823
I20250626 19:57:45.423611 5991 raft_consensus.cc:513] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:45.426007 5991 leader_election.cc:290] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:45.430275 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9172cb6576b240f6b25c3b1267618e6c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:45.432674 5992 raft_consensus.cc:491] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:45.433099 5992 raft_consensus.cc:513] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:45.433888 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9172cb6576b240f6b25c3b1267618e6c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:45.434445 4820 raft_consensus.cc:2466] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:45.435811 4609 leader_election.cc:304] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:45.436749 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69bd106066b6492bb7ea03d15a18df00" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:45.437417 5410 tablet_bootstrap.cc:492] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:45.438356 5992 leader_election.cc:290] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:45.439190 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:45.440264 4399 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (table default.Table_8b997467fa764e8fbf273c3b1f2ccb3e [id=760c56838ee140d78aa368a3810d19a8]). Expected version 1 got 0
I20250626 19:57:45.440305 5965 ts_tablet_manager.cc:1905] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:45.440635 5887 raft_consensus.cc:2802] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:45.440994 5887 raft_consensus.cc:491] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:45.441313 5887 raft_consensus.cc:3058] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:45.447074 4611 leader_election.cc:343] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:45.448122 5991 consensus_queue.cc:1035] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:45.453603 5887 raft_consensus.cc:513] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:45.496570 5887 leader_election.cc:290] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:45.449348 5353 tablet_bootstrap.cc:492] T ef1abf340cc34e00aa67e4588c61c32d P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:45.499912 5353 ts_tablet_manager.cc:1397] T ef1abf340cc34e00aa67e4588c61c32d P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.334s user 0.010s sys 0.001s
I20250626 19:57:45.500010 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9172cb6576b240f6b25c3b1267618e6c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:45.500475 4820 raft_consensus.cc:3058] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:45.500857 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_65406953a5444c54a77eaed676d03f6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:45.502357 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65406953a5444c54a77eaed676d03f6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:45.508216 5410 tablet_bootstrap.cc:654] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:45.509550 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9172cb6576b240f6b25c3b1267618e6c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:45.483194 5924 consensus_queue.cc:1035] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20250626 19:57:45.511209 4977 raft_consensus.cc:3058] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:45.511263 4611 leader_election.cc:343] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:45.515941 5937 ts_tablet_manager.cc:1918] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:45.523561 5937 log.cc:1199] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/f6ec952321924098a142f1438a283aab
I20250626 19:57:45.535465 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:45.537832 5353 raft_consensus.cc:357] T ef1abf340cc34e00aa67e4588c61c32d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:45.538482 5353 raft_consensus.cc:383] T ef1abf340cc34e00aa67e4588c61c32d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:45.551136 4820 raft_consensus.cc:2466] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
W20250626 19:57:45.554963 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 416028us
I20250626 19:57:45.559933 4391 master_service.cc:948] ReplaceTablet: received request to replace tablet 1feb6785d0c7459fbc6898b0c9d3e38c from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:45.578176 5936 ts_tablet_manager.cc:1918] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:45.583490 5936 log.cc:1199] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/f6ec952321924098a142f1438a283aab
I20250626 19:57:45.603544 5366 tablet_bootstrap.cc:654] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:45.610940 5936 ts_tablet_manager.cc:1939] T f6ec952321924098a142f1438a283aab P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:45.633721 4390 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:45.630489 5936 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 154) took 1314 ms. Trace:
I20250626 19:57:45.641496 5936 rpcz_store.cc:276] 0626 19:57:44.316583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.316680 (+ 97us) service_pool.cc:225] Handling call
0626 19:57:44.923790 (+607110us) ts_tablet_manager.cc:1145] Deleting tablet f6ec952321924098a142f1438a283aab
0626 19:57:44.923840 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:45.374889 (+451049us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:45.577901 (+203012us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:45.630418 (+ 52517us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":606553}
I20250626 19:57:45.643517 5936 tablet_replica.cc:331] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:45.643793 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet f6ec952321924098a142f1438a283aab (table default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]) successfully deleted
I20250626 19:57:45.644421 5936 raft_consensus.cc:2241] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:45.645288 5936 raft_consensus.cc:2270] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:45.647122 4390 catalog_manager.cc:5582] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:45.659735 4390 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 7851e464fcd44ecb82404d6a991db2a2 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]). Expected version 1 got 0
I20250626 19:57:45.663012 4390 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:45.663823 4390 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:45.666814 4390 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:45.673219 4390 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:45.676977 4390 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (table default.Table_8b997467fa764e8fbf273c3b1f2ccb3e [id=760c56838ee140d78aa368a3810d19a8]). Expected version 1 got 0
I20250626 19:57:45.538714 5353 raft_consensus.cc:738] T ef1abf340cc34e00aa67e4588c61c32d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:45.679531 5353 consensus_queue.cc:260] T ef1abf340cc34e00aa67e4588c61c32d P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:45.715092 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69bd106066b6492bb7ea03d15a18df00" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:45.715510 4393 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34796 (request call id 149) took 1763 ms. Trace:
I20250626 19:57:45.715919 4393 rpcz_store.cc:276] 0626 19:57:43.952076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.116503 (+164427us) service_pool.cc:225] Handling call
0626 19:57:44.117552 (+ 1049us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:44.118249 (+ 697us) catalog_manager.cc:3683] Apply alter schema
0626 19:57:44.118811 (+ 562us) catalog_manager.cc:3715] Acquired catalog manager lock
0626 19:57:44.119032 (+ 221us) catalog_manager.cc:3873] Updating metadata on disk
0626 19:57:44.310861 (+191829us) catalog_manager.cc:3904] Committing alterations to in-memory state
0626 19:57:45.715433 (+1404572us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:44.175264 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:44.175731 (+ 467us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:44.175759 (+ 28us) write_op.cc:435] Acquired schema lock
0626 19:57:44.175775 (+ 16us) tablet.cc:660] Decoding operations
0626 19:57:44.176054 (+ 279us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:44.176089 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:44.176112 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:44.176261 (+ 149us) tablet.cc:699] Row locks acquired
0626 19:57:44.176274 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:57:44.176389 (+ 115us) write_op.cc:270] Start()
0626 19:57:44.176474 (+ 85us) write_op.cc:276] Timestamp: P: 1750967864176434 usec, L: 0
0626 19:57:44.176493 (+ 19us) op_driver.cc:348] REPLICATION: starting
0626 19:57:44.177022 (+ 529us) log.cc:844] Serialized 405 byte log entry
0626 19:57:44.305666 (+128644us) op_driver.cc:464] REPLICATION: finished
0626 19:57:44.308066 (+ 2400us) write_op.cc:301] APPLY: starting
0626 19:57:44.308122 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:44.308208 (+ 86us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:44.308283 (+ 75us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:44.308445 (+ 162us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:44.308583 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:44.308629 (+ 46us) write_op.cc:312] APPLY: finished
0626 19:57:44.308789 (+ 160us) log.cc:844] Serialized 23 byte log entry
0626 19:57:44.308932 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.308991 (+ 59us) write_op.cc:454] Released schema lock
0626 19:57:44.309022 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:44.309234 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.309256 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2250,"apply.run_cpu_time_us":1141,"apply.run_wall_time_us":1140,"mutex_wait_us":51184,"num_ops":1,"prepare.queue_time_us":53703,"prepare.run_cpu_time_us":2079,"prepare.run_wall_time_us":3519,"raft.queue_time_us":565,"raft.run_cpu_time_us":637,"raft.run_wall_time_us":1372,"replication_time_us":129079}]]}
I20250626 19:57:45.726321 4393 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_d849d42d83a746bcabb98ca420c0ec24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:45.727762 5353 ts_tablet_manager.cc:1428] T ef1abf340cc34e00aa67e4588c61c32d P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.228s user 0.005s sys 0.000s
I20250626 19:57:45.730978 5366 tablet_bootstrap.cc:492] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:45.731400 5366 ts_tablet_manager.cc:1397] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.397s user 0.012s sys 0.000s
I20250626 19:57:45.731446 4609 leader_election.cc:304] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
W20250626 19:57:45.738386 4393 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d849d42d83a746bcabb98ca420c0ec24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:45.739939 4394 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 3cc96a3e9c174cbca0196c5a1b27455c
I20250626 19:57:45.742403 4394 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34796 (request call id 151) took 1729 ms. Trace:
I20250626 19:57:45.740310 4398 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 1feb6785d0c7459fbc6898b0c9d3e38c
I20250626 19:57:45.747339 5353 tablet_bootstrap.cc:492] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
W20250626 19:57:45.747416 4397 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9f09f0381eb420ca57c18a5be6f2cc2 consensus state has no leader
I20250626 19:57:45.747502 4398 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34796 (request call id 146) took 4694 ms. Trace:
I20250626 19:57:45.747892 4397 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34796 (request call id 150) took 1767 ms. Trace:
I20250626 19:57:45.747785 4398 rpcz_store.cc:276] 0626 19:57:41.053563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.061853 (+3008290us) service_pool.cc:225] Handling call
0626 19:57:44.063315 (+ 1462us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:44.063947 (+ 632us) catalog_manager.cc:2790] Modifying in-memory table state
0626 19:57:44.064074 (+ 127us) catalog_manager.cc:2799] Locking tablets
0626 19:57:44.064592 (+ 518us) catalog_manager.cc:2813] Removing table and tablets from system table
0626 19:57:44.181384 (+116792us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.181408 (+ 24us) write_op.cc:454] Released schema lock
0626 19:57:44.181648 (+ 240us) catalog_manager.cc:2831] Removing table from by-name map
0626 19:57:44.186814 (+ 5166us) catalog_manager.cc:2847] Committing in-memory state
0626 19:57:45.740240 (+1553426us) catalog_manager.cc:2851] Aborting table tasks
0626 19:57:45.747451 (+ 7211us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:44.121886 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:44.122336 (+ 450us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:44.122362 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:44.122378 (+ 16us) tablet.cc:660] Decoding operations
0626 19:57:44.122792 (+ 414us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:44.122846 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:44.122872 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:44.123160 (+ 288us) tablet.cc:699] Row locks acquired
0626 19:57:44.123179 (+ 19us) write_op.cc:260] PREPARE: finished
0626 19:57:44.123300 (+ 121us) write_op.cc:270] Start()
0626 19:57:44.123394 (+ 94us) write_op.cc:276] Timestamp: P: 1750967864123321 usec, L: 0
0626 19:57:44.123414 (+ 20us) op_driver.cc:348] REPLICATION: starting
0626 19:57:44.123980 (+ 566us) log.cc:844] Serialized 728 byte log entry
0626 19:57:44.146023 (+ 22043us) op_driver.cc:464] REPLICATION: finished
0626 19:57:44.146242 (+ 219us) write_op.cc:301] APPLY: starting
0626 19:57:44.146301 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:44.146419 (+ 118us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:44.146498 (+ 79us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:44.146813 (+ 315us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:44.146958 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:44.150825 (+ 3867us) write_op.cc:312] APPLY: finished
0626 19:57:44.151028 (+ 203us) log.cc:844] Serialized 29 byte log entry
0626 19:57:44.151174 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.151237 (+ 63us) write_op.cc:454] Released schema lock
0626 19:57:44.151271 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":75,"apply.run_cpu_time_us":1396,"apply.run_wall_time_us":5196,"num_ops":2,"prepare.queue_time_us":55700,"prepare.run_cpu_time_us":2394,"prepare.run_wall_time_us":2402,"raft.queue_time_us":6074,"raft.run_cpu_time_us":843,"raft.run_wall_time_us":842,"replication_time_us":22502}]]}
I20250626 19:57:45.752239 4398 master_service.cc:948] ReplaceTablet: received request to replace tablet 9172cb6576b240f6b25c3b1267618e6c from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:45.752696 5936 ts_tablet_manager.cc:1905] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:45.754874 4951 tablet_service.cc:1515] Processing DeleteTablet for tablet 1feb6785d0c7459fbc6898b0c9d3e38c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:45.753728 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cc96a3e9c174cbca0196c5a1b27455c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:45.756043 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cc96a3e9c174cbca0196c5a1b27455c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:45.742707 4394 rpcz_store.cc:276] 0626 19:57:44.012944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.241733 (+228789us) service_pool.cc:225] Handling call
0626 19:57:44.301671 (+ 59938us) spinlock_profiling.cc:243] Waited 59.3 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe791557046 00007fe78f2781a4 00007fe78f278132 00007fe78f278061 00007fe78f277fdc 00007fe78f277292 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:44.302706 (+ 1035us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:44.303201 (+ 495us) catalog_manager.cc:2790] Modifying in-memory table state
0626 19:57:44.303331 (+ 130us) catalog_manager.cc:2799] Locking tablets
0626 19:57:44.303878 (+ 547us) catalog_manager.cc:2813] Removing table and tablets from system table
0626 19:57:44.460731 (+156853us) catalog_manager.cc:2831] Removing table from by-name map
0626 19:57:44.489030 (+ 28299us) catalog_manager.cc:2847] Committing in-memory state
0626 19:57:45.739834 (+1250804us) catalog_manager.cc:2851] Aborting table tasks
0626 19:57:45.742341 (+ 2507us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:44.367855 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:44.368308 (+ 453us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:44.368334 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:44.368353 (+ 19us) tablet.cc:660] Decoding operations
0626 19:57:44.368692 (+ 339us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:44.368737 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:44.368761 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:44.369037 (+ 276us) tablet.cc:699] Row locks acquired
0626 19:57:44.369055 (+ 18us) write_op.cc:260] PREPARE: finished
0626 19:57:44.369161 (+ 106us) write_op.cc:270] Start()
0626 19:57:44.369250 (+ 89us) write_op.cc:276] Timestamp: P: 1750967864369282 usec, L: 0
0626 19:57:44.369268 (+ 18us) op_driver.cc:348] REPLICATION: starting
0626 19:57:44.418723 (+ 49455us) log.cc:844] Serialized 768 byte log entry
0626 19:57:44.425461 (+ 6738us) spinlock_profiling.cc:243] Waited 48.9 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72b481 00007fe78f7a2a18 00007fe78fc0f811 00007fe78fc0efe8 00007fe78fc13c81 00007fe78fc13c39 00007fe78fc13bc9 00007fe78fc13b91 00007fe78fc12dbd 000000000059bcd4 000000000059bb09
0626 19:57:44.444166 (+ 18705us) op_driver.cc:464] REPLICATION: finished
0626 19:57:44.446892 (+ 2726us) write_op.cc:301] APPLY: starting
0626 19:57:44.447345 (+ 453us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:44.448195 (+ 850us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:44.448491 (+ 296us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:44.452899 (+ 4408us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:44.454103 (+ 1204us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:44.454541 (+ 438us) write_op.cc:312] APPLY: finished
0626 19:57:44.456499 (+ 1958us) log.cc:844] Serialized 29 byte log entry
0626 19:57:44.458309 (+ 1810us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.459257 (+ 948us) write_op.cc:454] Released schema lock
0626 19:57:44.459719 (+ 462us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:44.464372 (+ 4653us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.464989 (+ 617us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":136391424,"child_traces":[["op",{"apply.queue_time_us":1388,"apply.run_cpu_time_us":2998,"apply.run_wall_time_us":15796,"num_ops":2,"prepare.queue_time_us":62324,"prepare.run_cpu_time_us":3720,"prepare.run_wall_time_us":57827,"raft.queue_time_us":203,"raft.run_cpu_time_us":559,"raft.run_wall_time_us":559,"replication_time_us":74594,"spinlock_wait_cycles":112441600}]]}
I20250626 19:57:45.756665 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 1feb6785d0c7459fbc6898b0c9d3e38c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:45.758647 4394 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_3bffc9f08d164dafb33290412ccbd109"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:45.760370 4394 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bffc9f08d164dafb33290412ccbd109 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:45.748155 4397 rpcz_store.cc:276] 0626 19:57:43.980921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.158661 (+177740us) service_pool.cc:225] Handling call
0626 19:57:44.159576 (+ 915us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:44.211235 (+ 51659us) catalog_manager.cc:3683] Apply alter schema
0626 19:57:44.211792 (+ 557us) catalog_manager.cc:3715] Acquired catalog manager lock
0626 19:57:44.211995 (+ 203us) catalog_manager.cc:3873] Updating metadata on disk
0626 19:57:44.441045 (+229050us) catalog_manager.cc:3904] Committing alterations to in-memory state
0626 19:57:45.747843 (+1306798us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:44.293608 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:44.294102 (+ 494us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:44.294133 (+ 31us) write_op.cc:435] Acquired schema lock
0626 19:57:44.294152 (+ 19us) tablet.cc:660] Decoding operations
0626 19:57:44.294491 (+ 339us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:44.294545 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:44.294575 (+ 30us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:44.294769 (+ 194us) tablet.cc:699] Row locks acquired
0626 19:57:44.294786 (+ 17us) write_op.cc:260] PREPARE: finished
0626 19:57:44.294905 (+ 119us) write_op.cc:270] Start()
0626 19:57:44.294993 (+ 88us) write_op.cc:276] Timestamp: P: 1750967864294994 usec, L: 0
0626 19:57:44.295015 (+ 22us) op_driver.cc:348] REPLICATION: starting
0626 19:57:44.295556 (+ 541us) log.cc:844] Serialized 405 byte log entry
0626 19:57:44.438004 (+142448us) op_driver.cc:464] REPLICATION: finished
0626 19:57:44.438534 (+ 530us) write_op.cc:301] APPLY: starting
0626 19:57:44.438590 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:44.438678 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:44.438838 (+ 160us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:44.439011 (+ 173us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:44.439136 (+ 125us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:44.439178 (+ 42us) write_op.cc:312] APPLY: finished
0626 19:57:44.439343 (+ 165us) log.cc:844] Serialized 23 byte log entry
0626 19:57:44.439484 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.439536 (+ 52us) write_op.cc:454] Released schema lock
0626 19:57:44.439565 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:44.439860 (+ 295us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.439895 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":264,"apply.run_cpu_time_us":1239,"apply.run_wall_time_us":1314,"num_ops":1,"prepare.queue_time_us":80054,"prepare.run_cpu_time_us":2359,"prepare.run_wall_time_us":74131,"raft.queue_time_us":4763,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":238,"replication_time_us":142705}]]}
I20250626 19:57:45.762151 4397 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_d9f785633e4f45eaa38fd0ec36f46224"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:45.763774 4397 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9f785633e4f45eaa38fd0ec36f46224 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:45.759752 5965 ts_tablet_manager.cc:1918] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:45.777251 5937 ts_tablet_manager.cc:1939] T f6ec952321924098a142f1438a283aab P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:45.774140 5887 raft_consensus.cc:491] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:45.779810 5965 log.cc:1199] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/f6ec952321924098a142f1438a283aab
I20250626 19:57:45.783421 5996 raft_consensus.cc:2802] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:45.783897 5996 raft_consensus.cc:695] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:45.779881 5887 raft_consensus.cc:513] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:45.784628 5996 consensus_queue.cc:237] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:45.794761 4951 tablet_service.cc:1515] Processing DeleteTablet for tablet fc526a05990446a3b3bc973e79511476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:45.795564 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b66cda4f8ed410184b434d5083deac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7851e464fcd44ecb82404d6a991db2a2) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:45.795799 5887 leader_election.cc:290] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:45.790470 5937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 165) took 1475 ms. Trace:
W20250626 19:57:45.796839 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 65514.7us
I20250626 19:57:45.797611 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:45.801220 4951 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:45.801733 5887 consensus_queue.cc:1035] T 781f8ee8c8ad49c6b2391fcdf20bdfdb P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:45.801952 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef1abf340cc34e00aa67e4588c61c32d" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:45.803591 4816 raft_consensus.cc:2466] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:45.797175 5937 rpcz_store.cc:276] 0626 19:57:44.315394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.315588 (+ 194us) service_pool.cc:225] Handling call
0626 19:57:45.154473 (+838885us) ts_tablet_manager.cc:1145] Deleting tablet f6ec952321924098a142f1438a283aab
0626 19:57:45.154522 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:45.390954 (+236432us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:45.508395 (+117441us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:45.790385 (+281990us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":838329}
I20250626 19:57:45.804713 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet 609d36d46bb84017b2a3a7ad48bfccfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:45.804875 5937 tablet_replica.cc:331] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:45.805517 5965 ts_tablet_manager.cc:1939] T f6ec952321924098a142f1438a283aab P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
W20250626 19:57:45.805981 4395 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 66) took 4679 ms (client timeout 1000 ms). Trace:
I20250626 19:57:45.806973 5026 heartbeater.cc:344] Connected to a master server at 127.4.46.190:33823
W20250626 19:57:45.806306 4395 rpcz_store.cc:269] 0626 19:57:41.126593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:41.393414 (+266821us) service_pool.cc:225] Handling call
0626 19:57:45.805907 (+4412493us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:43.824374 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:43.824811 (+ 437us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:43.824834 (+ 23us) write_op.cc:435] Acquired schema lock
0626 19:57:43.824845 (+ 11us) tablet.cc:660] Decoding operations
0626 19:57:43.825113 (+ 268us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:43.825170 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:43.825195 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:43.825353 (+ 158us) tablet.cc:699] Row locks acquired
0626 19:57:43.825365 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:43.825488 (+ 123us) write_op.cc:270] Start()
0626 19:57:43.825576 (+ 88us) write_op.cc:276] Timestamp: P: 1750967863825592 usec, L: 0
0626 19:57:43.825597 (+ 21us) op_driver.cc:348] REPLICATION: starting
0626 19:57:43.826293 (+ 696us) log.cc:844] Serialized 530 byte log entry
0626 19:57:43.957470 (+131177us) op_driver.cc:464] REPLICATION: finished
0626 19:57:44.013982 (+ 56512us) write_op.cc:301] APPLY: starting
0626 19:57:44.014044 (+ 62us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:44.014138 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:44.014231 (+ 93us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:44.014469 (+ 238us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:44.014670 (+ 201us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:44.014740 (+ 70us) write_op.cc:312] APPLY: finished
0626 19:57:44.019765 (+ 5025us) log.cc:844] Serialized 23 byte log entry
0626 19:57:44.019952 (+ 187us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.020016 (+ 64us) write_op.cc:454] Released schema lock
0626 19:57:44.020056 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:44.142079 (+122023us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.142106 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":57,"child_traces":[["op",{"apply.queue_time_us":56322,"apply.run_cpu_time_us":1593,"apply.run_wall_time_us":128074,"num_ops":1,"prepare.queue_time_us":16073,"prepare.run_cpu_time_us":2316,"prepare.run_wall_time_us":11985,"raft.queue_time_us":114,"raft.run_cpu_time_us":790,"raft.run_wall_time_us":790,"replication_time_us":131761,"thread_start_us":94501,"threads_started":2}]]}
I20250626 19:57:45.809890 4609 leader_election.cc:304] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:45.811280 5991 raft_consensus.cc:2802] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:45.811651 5991 raft_consensus.cc:491] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 19:57:45.812166 4611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 80006.3us
I20250626 19:57:45.812757 5991 raft_consensus.cc:3058] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:45.811655 5937 raft_consensus.cc:2241] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:45.820178 6014 consensus_queue.cc:1035] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:45.825121 5937 raft_consensus.cc:2270] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:45.828405 5937 ts_tablet_manager.cc:1905] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:45.830729 5991 raft_consensus.cc:513] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:45.832505 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef1abf340cc34e00aa67e4588c61c32d" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:45.832787 5991 leader_election.cc:290] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:45.833133 4979 raft_consensus.cc:2466] T ef1abf340cc34e00aa67e4588c61c32d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:45.835784 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet f6ec952321924098a142f1438a283aab (table default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]) successfully deleted
I20250626 19:57:45.837216 4654 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cc96a3e9c174cbca0196c5a1b27455c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:45.837924 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 1feb6785d0c7459fbc6898b0c9d3e38c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:45.840781 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef1abf340cc34e00aa67e4588c61c32d" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:45.841315 4977 raft_consensus.cc:3058] T ef1abf340cc34e00aa67e4588c61c32d P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:45.848577 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef1abf340cc34e00aa67e4588c61c32d" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:45.849140 4817 raft_consensus.cc:3058] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:45.850989 4977 raft_consensus.cc:2466] T ef1abf340cc34e00aa67e4588c61c32d P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:45.875790 5993 consensus_queue.cc:1035] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:45.881798 4611 leader_election.cc:304] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:45.882956 6015 raft_consensus.cc:2802] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:45.887691 5965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 172) took 1571 ms. Trace:
I20250626 19:57:45.888178 5965 rpcz_store.cc:276] 0626 19:57:44.316719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.316902 (+ 183us) service_pool.cc:225] Handling call
0626 19:57:45.400741 (+1083839us) ts_tablet_manager.cc:1145] Deleting tablet f6ec952321924098a142f1438a283aab
0626 19:57:45.400793 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:45.570717 (+169924us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:45.710957 (+140240us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:45.887609 (+176652us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1083301}
I20250626 19:57:45.889659 5965 tablet_replica.cc:331] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
W20250626 19:57:45.892906 4383 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 7851e464fcd44ecb82404d6a991db2a2 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7851e464fcd44ecb82404d6a991db2a2 consensus state has no leader
I20250626 19:57:45.895097 5965 raft_consensus.cc:2241] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:45.895774 5965 raft_consensus.cc:2270] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:45.898319 5965 ts_tablet_manager.cc:1905] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:45.899459 5366 raft_consensus.cc:357] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:45.900219 5366 raft_consensus.cc:383] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:45.885435 6015 raft_consensus.cc:695] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:45.907974 6015 consensus_queue.cc:237] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:45.910673 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 44295.8us
I20250626 19:57:45.912266 5992 consensus_queue.cc:1035] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:45.916400 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet f6ec952321924098a142f1438a283aab (table default.Table_605c8b4275614ae2a45a5ad6224f3a43 [id=d0568845739d4a53a9dfc5fb6da2d4a0]) successfully deleted
I20250626 19:57:45.929030 5353 tablet_bootstrap.cc:654] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:45.933264 5366 raft_consensus.cc:738] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:45.934010 5366 consensus_queue.cc:260] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:45.936353 5366 ts_tablet_manager.cc:1428] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.205s user 0.005s sys 0.000s
I20250626 19:57:45.945796 5998 raft_consensus.cc:491] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:45.946297 5998 raft_consensus.cc:513] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:45.948194 5998 leader_election.cc:290] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:45.949822 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9f09f0381eb420ca57c18a5be6f2cc2" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:45.950490 4396 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 7851e464fcd44ecb82404d6a991db2a2 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 7851e464fcd44ecb82404d6a991db2a2 consensus state has no leader
W20250626 19:57:45.951942 4755 leader_election.cc:343] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:45.952136 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c0338c05a974e0f96865398722c5e8f with delete_type TABLET_DATA_DELETED (replaced by tablet 34b2919e5f534f4295336a4384be1f64) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:45.954324 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9f09f0381eb420ca57c18a5be6f2cc2" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:45.956266 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet f6ec952321924098a142f1438a283aab with delete_type TABLET_DATA_DELETED (replaced by tablet 2d0a30ee16d043f9837dce612dae670a) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:45.968291 4394 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_e6347deb7eb14f08bd5e02821a5a55dd" } new_table_name: "default.Table_56852b6e38af4545879ff672cc528975" modify_external_catalogs: true
I20250626 19:57:45.970577 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet fc526a05990446a3b3bc973e79511476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:45.972023 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:45.973631 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:45.974983 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 609d36d46bb84017b2a3a7ad48bfccfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:40742
W20250626 19:57:45.976622 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 61751.3us
I20250626 19:57:45.977257 4911 leader_election.cc:304] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:45.979053 6027 raft_consensus.cc:2747] T 69bd106066b6492bb7ea03d15a18df00 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:45.981449 4611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 99795.4us
I20250626 19:57:45.984625 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:45.987147 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_714f173170274ff7ac112d613b731467"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:45.988721 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_714f173170274ff7ac112d613b731467 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:45.995400 5366 tablet_bootstrap.cc:492] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:46.000849 4399 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: default.Table_4708ce225be445acbb30bbcc243949bc [id=ae81e1731bc1461385bf84e443ff0730] alter complete (version 1)
W20250626 19:57:46.002213 4756 leader_election.cc:343] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:46.002605 4756 leader_election.cc:304] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:46.003705 6010 raft_consensus.cc:2747] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:46.016662 4397 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_822bad36b81c42ac9101bcab4aaca9a4" } modify_external_catalogs: true
I20250626 19:57:45.998602 5410 tablet_bootstrap.cc:492] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:46.017459 4397 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_822bad36b81c42ac9101bcab4aaca9a4" } modify_external_catalogs: true
I20250626 19:57:46.017935 5410 ts_tablet_manager.cc:1397] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.581s user 0.012s sys 0.000s
I20250626 19:57:46.021003 5410 raft_consensus.cc:357] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.021636 5410 raft_consensus.cc:383] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:46.021910 5410 raft_consensus.cc:738] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:46.028589 4817 raft_consensus.cc:2466] T ef1abf340cc34e00aa67e4588c61c32d P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
W20250626 19:57:46.032586 4396 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet de0724ba71824a5daaf6f3708b4431b7 consensus state has no leader
W20250626 19:57:46.034727 4396 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 78) took 2000 ms (client timeout 1000 ms). Trace:
I20250626 19:57:46.040611 5353 tablet_bootstrap.cc:492] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:46.041067 5353 ts_tablet_manager.cc:1397] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.294s user 0.005s sys 0.003s
I20250626 19:57:46.056820 6032 tablet.cc:1722] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:46.059787 6034 tablet.cc:1722] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:46.070108 4398 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 9172cb6576b240f6b25c3b1267618e6c
I20250626 19:57:46.071374 4398 catalog_manager.cc:6652] ReplaceTablet: tablet 9172cb6576b240f6b25c3b1267618e6c deleted and replaced by tablet 290232b0933b451c9d7744165bd72796
I20250626 19:57:46.072064 4398 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_8ef072e9d240402397d79db39729e650" } modify_external_catalogs: true
I20250626 19:57:46.072525 4398 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_8ef072e9d240402397d79db39729e650" } modify_external_catalogs: true
I20250626 19:57:46.073904 4952 tablet_service.cc:1515] Processing DeleteTablet for tablet 9172cb6576b240f6b25c3b1267618e6c with delete_type TABLET_DATA_DELETED (replaced by tablet 290232b0933b451c9d7744165bd72796) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:46.073948 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 9172cb6576b240f6b25c3b1267618e6c with delete_type TABLET_DATA_DELETED (replaced by tablet 290232b0933b451c9d7744165bd72796) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:46.077991 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 9172cb6576b240f6b25c3b1267618e6c with delete_type TABLET_DATA_DELETED (replaced by tablet 290232b0933b451c9d7744165bd72796) from {username='slave'} at 127.0.0.1:40742
W20250626 19:57:46.035161 4396 rpcz_store.cc:269] 0626 19:57:44.034100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.043035 (+ 8935us) service_pool.cc:225] Handling call
0626 19:57:44.353245 (+310210us) spinlock_profiling.cc:243] Waited 29.9 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe7915557ee 00007fe78f26c624 00007fe78f26c5b2 00007fe78f26c4e1 00007fe78f26c45c 00007fe78f26b712 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:46.034661 (+1681416us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:44.762310 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:44.762814 (+ 504us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:44.762840 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:44.762856 (+ 16us) tablet.cc:660] Decoding operations
0626 19:57:44.764656 (+ 1800us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:44.764697 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:44.764724 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:44.764888 (+ 164us) tablet.cc:699] Row locks acquired
0626 19:57:44.764905 (+ 17us) write_op.cc:260] PREPARE: finished
0626 19:57:44.765026 (+ 121us) write_op.cc:270] Start()
0626 19:57:44.765120 (+ 94us) write_op.cc:276] Timestamp: P: 1750967864765103 usec, L: 0
0626 19:57:44.765140 (+ 20us) op_driver.cc:348] REPLICATION: starting
0626 19:57:44.765696 (+ 556us) log.cc:844] Serialized 530 byte log entry
0626 19:57:44.796589 (+ 30893us) op_driver.cc:464] REPLICATION: finished
0626 19:57:44.798901 (+ 2312us) write_op.cc:301] APPLY: starting
0626 19:57:44.798974 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:44.799062 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:44.799154 (+ 92us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:44.799328 (+ 174us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:44.799450 (+ 122us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:44.799495 (+ 45us) write_op.cc:312] APPLY: finished
0626 19:57:44.799653 (+ 158us) log.cc:844] Serialized 23 byte log entry
0626 19:57:44.799799 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.799857 (+ 58us) write_op.cc:454] Released schema lock
0626 19:57:44.799886 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:44.800108 (+ 222us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.800132 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20062,"reported_tablets":73,"spinlock_wait_cycles":68800256,"child_traces":[["op",{"apply.queue_time_us":2159,"apply.run_cpu_time_us":1176,"apply.run_wall_time_us":1175,"num_ops":1,"prepare.queue_time_us":1953,"prepare.run_cpu_time_us":2171,"prepare.run_wall_time_us":3629,"replication_time_us":31354}]]}
I20250626 19:57:46.087419 6026 consensus_queue.cc:1035] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:46.079842 5410 consensus_queue.cc:260] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.095643 5353 raft_consensus.cc:357] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.096505 5353 raft_consensus.cc:383] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:46.096818 5353 raft_consensus.cc:738] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:46.097508 5353 consensus_queue.cc:260] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.098287 5410 ts_tablet_manager.cc:1428] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.080s user 0.006s sys 0.000s
I20250626 19:57:46.099294 5410 tablet_bootstrap.cc:492] T 125f7265258e48368381aa54559eb510 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:46.104651 6010 raft_consensus.cc:491] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:46.105253 6010 raft_consensus.cc:513] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:46.109112 6010 leader_election.cc:290] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:46.109925 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fafbf98e7f4544b6ad96c1c5ab6c82bc" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:46.111378 4755 leader_election.cc:343] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:46.115823 5366 tablet_bootstrap.cc:654] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:46.124881 4394 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7def0bb1dbdc494eaa71748ebe1f8847 consensus state has no leader
I20250626 19:57:46.124742 6036 tablet.cc:1722] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:46.125535 4396 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_c8720865c8c742bc941a7d2665c618f5" } new_table_name: "default.Table_2d1ec663bb8048e0ac15d7025383ef5d" modify_external_catalogs: true
I20250626 19:57:46.135177 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fafbf98e7f4544b6ad96c1c5ab6c82bc" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:46.155962 4756 leader_election.cc:343] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:46.156484 4756 leader_election.cc:304] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:46.172632 5353 ts_tablet_manager.cc:1428] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.131s user 0.005s sys 0.002s
I20250626 19:57:46.172967 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_0bdf32dfb89d4500877aeed0695d926f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:46.173590 5353 tablet_bootstrap.cc:492] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:46.175743 4394 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_90bb38ee5f4c4b3ca1d9f018e53ef6b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:46.185019 6010 raft_consensus.cc:2747] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:46.199151 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c0338c05a974e0f96865398722c5e8f with delete_type TABLET_DATA_DELETED (replaced by tablet 34b2919e5f534f4295336a4384be1f64) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:46.216075 5937 ts_tablet_manager.cc:1918] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:46.216493 5937 log.cc:1199] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/65af5ae255d74143976392bccc50f3ba
I20250626 19:57:46.227516 5366 tablet_bootstrap.cc:492] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:46.228000 5366 ts_tablet_manager.cc:1397] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.233s user 0.010s sys 0.000s
W20250626 19:57:46.229727 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bdf32dfb89d4500877aeed0695d926f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:57:46.230599 4394 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90bb38ee5f4c4b3ca1d9f018e53ef6b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:46.231911 5366 raft_consensus.cc:357] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.232555 5366 raft_consensus.cc:383] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:46.232851 5366 raft_consensus.cc:738] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:46.233522 5366 consensus_queue.cc:260] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.218828 5936 ts_tablet_manager.cc:1918] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:46.234532 5936 log.cc:1199] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/65af5ae255d74143976392bccc50f3ba
I20250626 19:57:46.237712 4654 tablet_service.cc:1515] Processing DeleteTablet for tablet 36997dab0ab3430bba63a90805759b30 with delete_type TABLET_DATA_DELETED (replaced by tablet fafbf98e7f4544b6ad96c1c5ab6c82bc) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:46.238772 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet fc526a05990446a3b3bc973e79511476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:46.239358 4650 tablet_service.cc:1515] Processing DeleteTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:46.240072 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:46.240890 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet f6ec952321924098a142f1438a283aab with delete_type TABLET_DATA_DELETED (replaced by tablet 2d0a30ee16d043f9837dce612dae670a) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:46.239367 4654 tablet_service.cc:1515] Processing DeleteTablet for tablet 609d36d46bb84017b2a3a7ad48bfccfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:46.241768 5887 raft_consensus.cc:491] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:46.242216 5887 raft_consensus.cc:513] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.244899 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8be634cb36d40dca3f9338dab93ad2c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:46.245581 5887 leader_election.cc:290] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:46.246702 5936 ts_tablet_manager.cc:1939] T 65af5ae255d74143976392bccc50f3ba P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:46.246730 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8be634cb36d40dca3f9338dab93ad2c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:46.247468 4820 raft_consensus.cc:2466] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
W20250626 19:57:46.248615 4611 leader_election.cc:343] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:46.251271 5937 ts_tablet_manager.cc:1939] T 65af5ae255d74143976392bccc50f3ba P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:46.252117 5937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 166) took 1306 ms. Trace:
I20250626 19:57:46.252444 5937 rpcz_store.cc:276] 0626 19:57:44.945750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.945889 (+ 139us) service_pool.cc:225] Handling call
0626 19:57:45.804694 (+858805us) ts_tablet_manager.cc:1145] Deleting tablet 65af5ae255d74143976392bccc50f3ba
0626 19:57:45.804754 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:45.977600 (+172846us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:46.215831 (+238231us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:46.252058 (+ 36227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":858193}
I20250626 19:57:46.253836 5937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 170) took 1225 ms. Trace:
I20250626 19:57:46.261641 5965 ts_tablet_manager.cc:1918] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:46.262097 5965 log.cc:1199] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/65af5ae255d74143976392bccc50f3ba
I20250626 19:57:46.272250 5353 tablet_bootstrap.cc:654] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:46.280802 5366 ts_tablet_manager.cc:1428] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
I20250626 19:57:46.253877 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 65af5ae255d74143976392bccc50f3ba (table default.Table_d2c793438d234509974deb53c4e61c8b [id=84b3a43b3e5b42849f8b343a97906309]) successfully deleted
W20250626 19:57:46.287918 4392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 82) took 3069 ms (client timeout 1000 ms). Trace:
I20250626 19:57:46.289675 5366 tablet_bootstrap.cc:492] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
W20250626 19:57:46.288357 4392 rpcz_store.cc:269] 0626 19:57:43.218649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:43.740911 (+522262us) service_pool.cc:225] Handling call
0626 19:57:46.287848 (+2546937us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:44.593761 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:44.594193 (+ 432us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:44.594223 (+ 30us) write_op.cc:435] Acquired schema lock
0626 19:57:44.594249 (+ 26us) tablet.cc:660] Decoding operations
0626 19:57:44.599269 (+ 5020us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:44.599315 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:44.599344 (+ 29us) tablet.cc:683] Acquiring locks for 9 operations
0626 19:57:44.599980 (+ 636us) tablet.cc:699] Row locks acquired
0626 19:57:44.599994 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:57:44.600116 (+ 122us) write_op.cc:270] Start()
0626 19:57:44.600204 (+ 88us) write_op.cc:276] Timestamp: P: 1750967864600130 usec, L: 0
0626 19:57:44.600223 (+ 19us) op_driver.cc:348] REPLICATION: starting
0626 19:57:44.608215 (+ 7992us) log.cc:844] Serialized 3547 byte log entry
0626 19:57:44.621303 (+ 13088us) op_driver.cc:464] REPLICATION: finished
0626 19:57:44.621579 (+ 276us) write_op.cc:301] APPLY: starting
0626 19:57:44.621637 (+ 58us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:44.621821 (+ 184us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:44.621912 (+ 91us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:44.622727 (+ 815us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:44.622924 (+ 197us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0626 19:57:44.622970 (+ 46us) write_op.cc:312] APPLY: finished
0626 19:57:44.623215 (+ 245us) log.cc:844] Serialized 71 byte log entry
0626 19:57:44.623358 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.623458 (+ 100us) write_op.cc:454] Released schema lock
0626 19:57:44.623497 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:44.623726 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:44.623751 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":85149,"reported_tablets":67,"child_traces":[["op",{"apply.queue_time_us":142,"apply.run_cpu_time_us":2106,"apply.run_wall_time_us":2104,"num_ops":9,"prepare.queue_time_us":1531,"prepare.run_cpu_time_us":3203,"prepare.run_wall_time_us":25096,"replication_time_us":20973}]]}
I20250626 19:57:46.291891 5965 ts_tablet_manager.cc:1939] T 65af5ae255d74143976392bccc50f3ba P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:46.292760 4392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_9357cbde46f74ae583f88b25120e4811"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:46.293969 5965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 173) took 1348 ms. Trace:
W20250626 19:57:46.302361 4392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9357cbde46f74ae583f88b25120e4811 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:46.254164 5937 rpcz_store.cc:276] 0626 19:57:45.028025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.028134 (+ 109us) service_pool.cc:225] Handling call
0626 19:57:46.253460 (+1225326us) ts_tablet_manager.cc:1145] Deleting tablet e94a73b50b564a97bbee0f5015bb8805
0626 19:57:46.253502 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:46.253783 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1224739}
I20250626 19:57:46.322405 5366 tablet_bootstrap.cc:654] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:46.311425 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet e94a73b50b564a97bbee0f5015bb8805 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e94a73b50b564a97bbee0f5015bb8805 already in progress: creating tablet
W20250626 19:57:46.326460 4755 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 68711.9us
I20250626 19:57:46.329615 5936 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 155) took 1382 ms. Trace:
I20250626 19:57:46.329953 5936 rpcz_store.cc:276] 0626 19:57:44.946936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.947057 (+ 121us) service_pool.cc:225] Handling call
0626 19:57:45.643338 (+696281us) ts_tablet_manager.cc:1145] Deleting tablet 65af5ae255d74143976392bccc50f3ba
0626 19:57:45.643396 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:46.063342 (+419946us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:46.218536 (+155194us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:46.329539 (+111003us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50653,"tablet-delete.queue_time_us":695750}
I20250626 19:57:46.332180 5936 tablet_replica.cc:331] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:46.333109 5936 raft_consensus.cc:2241] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:46.333629 5936 raft_consensus.cc:2270] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:46.335238 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 65af5ae255d74143976392bccc50f3ba (table default.Table_d2c793438d234509974deb53c4e61c8b [id=84b3a43b3e5b42849f8b343a97906309]) successfully deleted
I20250626 19:57:46.336469 5936 ts_tablet_manager.cc:1905] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:46.339784 5937 tablet_replica.cc:331] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:46.342588 5937 raft_consensus.cc:2241] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:46.357390 4392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_91248389db7e4324a3ddc4e5874b2666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:46.294333 5965 rpcz_store.cc:276] 0626 19:57:44.945530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.945684 (+ 154us) service_pool.cc:225] Handling call
0626 19:57:45.889446 (+943762us) ts_tablet_manager.cc:1145] Deleting tablet 65af5ae255d74143976392bccc50f3ba
0626 19:57:45.889504 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:46.014498 (+124994us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:46.261339 (+246841us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:46.293909 (+ 32570us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":943115}
I20250626 19:57:46.360401 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 65af5ae255d74143976392bccc50f3ba (table default.Table_d2c793438d234509974deb53c4e61c8b [id=84b3a43b3e5b42849f8b343a97906309]) successfully deleted
W20250626 19:57:46.362447 4392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91248389db7e4324a3ddc4e5874b2666 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:46.374076 4394 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_89c3869cb74248c4af47e8eaa3371b16" } new_table_name: "default.Table_6251ce018aaf4ee0948f7c26518cb21f" modify_external_catalogs: true
I20250626 19:57:46.374594 5965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 177) took 1346 ms. Trace:
I20250626 19:57:46.374989 5965 rpcz_store.cc:276] 0626 19:57:45.027989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.028151 (+ 162us) service_pool.cc:225] Handling call
0626 19:57:46.360063 (+1331912us) ts_tablet_manager.cc:1145] Deleting tablet e94a73b50b564a97bbee0f5015bb8805
0626 19:57:46.374218 (+ 14155us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:46.374530 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1331274}
W20250626 19:57:46.376356 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet e94a73b50b564a97bbee0f5015bb8805 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e94a73b50b564a97bbee0f5015bb8805 already in progress: creating tablet
I20250626 19:57:46.376824 5965 tablet_replica.cc:331] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
W20250626 19:57:46.377395 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 1feb6785d0c7459fbc6898b0c9d3e38c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1feb6785d0c7459fbc6898b0c9d3e38c already in progress: creating tablet
I20250626 19:57:46.377873 5965 raft_consensus.cc:2241] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:46.378417 5965 raft_consensus.cc:2270] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:46.382256 5924 raft_consensus.cc:491] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:46.382704 5924 raft_consensus.cc:513] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.383705 5937 raft_consensus.cc:2270] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:46.384696 5924 leader_election.cc:290] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:46.387028 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9172cb6576b240f6b25c3b1267618e6c" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:46.387861 4820 raft_consensus.cc:2391] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:46.397749 5366 tablet_bootstrap.cc:492] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:46.398255 5366 ts_tablet_manager.cc:1397] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.109s user 0.010s sys 0.000s
I20250626 19:57:46.400337 4952 tablet_service.cc:1468] Processing CreateTablet for tablet 8c5c000402db45e182a551cd80626665 (DEFAULT_TABLE table=default.Table_ec5124d21b3d42e294ed85b470fbe5e4 [id=e470594ce25244dea20936186e2cd2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.404161 5353 tablet_bootstrap.cc:492] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:46.404611 5353 ts_tablet_manager.cc:1397] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.231s user 0.007s sys 0.004s
I20250626 19:57:46.406515 4954 tablet_service.cc:1468] Processing CreateTablet for tablet 2999736d3ea4428b8ce9668aeef52823 (DEFAULT_TABLE table=default.Table_b7a7e36d615144f692ba3b32abc5fb5d [id=a0aac9825f554aa696cf54f2e28aeb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.406860 4795 tablet_service.cc:1468] Processing CreateTablet for tablet 8c5c000402db45e182a551cd80626665 (DEFAULT_TABLE table=default.Table_ec5124d21b3d42e294ed85b470fbe5e4 [id=e470594ce25244dea20936186e2cd2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.407819 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c5c000402db45e182a551cd80626665. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.407809 4951 tablet_service.cc:1468] Processing CreateTablet for tablet 8c4549e621504a30ba5be5b0f139c905 (DEFAULT_TABLE table=default.Table_54b68b0dd5ba469cba186a5d4c6a9941 [id=7f7b8aca635746e0868116318bd54c0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.412063 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 2999736d3ea4428b8ce9668aeef52823 (DEFAULT_TABLE table=default.Table_b7a7e36d615144f692ba3b32abc5fb5d [id=a0aac9825f554aa696cf54f2e28aeb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.413069 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2999736d3ea4428b8ce9668aeef52823. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.414336 4952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c5c000402db45e182a551cd80626665. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.419164 4954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2999736d3ea4428b8ce9668aeef52823. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.408283 5366 raft_consensus.cc:357] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.421968 5937 ts_tablet_manager.cc:1905] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:46.426903 5366 raft_consensus.cc:383] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:46.427317 4951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c4549e621504a30ba5be5b0f139c905. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.436890 5410 tablet_bootstrap.cc:654] T 125f7265258e48368381aa54559eb510 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:46.440519 4654 tablet_service.cc:1468] Processing CreateTablet for tablet 8c5c000402db45e182a551cd80626665 (DEFAULT_TABLE table=default.Table_ec5124d21b3d42e294ed85b470fbe5e4 [id=e470594ce25244dea20936186e2cd2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.441510 4654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c5c000402db45e182a551cd80626665. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.443455 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 2999736d3ea4428b8ce9668aeef52823 (DEFAULT_TABLE table=default.Table_b7a7e36d615144f692ba3b32abc5fb5d [id=a0aac9825f554aa696cf54f2e28aeb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.444381 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2999736d3ea4428b8ce9668aeef52823. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.447165 5353 raft_consensus.cc:357] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.447393 4652 tablet_service.cc:1468] Processing CreateTablet for tablet 8c4549e621504a30ba5be5b0f139c905 (DEFAULT_TABLE table=default.Table_54b68b0dd5ba469cba186a5d4c6a9941 [id=7f7b8aca635746e0868116318bd54c0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.447805 5353 raft_consensus.cc:383] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:46.448031 5353 raft_consensus.cc:738] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:46.448288 4652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c4549e621504a30ba5be5b0f139c905. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.449070 5353 consensus_queue.cc:260] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.427304 5366 raft_consensus.cc:738] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:46.451363 5366 consensus_queue.cc:260] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.456938 4957 tablet_service.cc:1468] Processing CreateTablet for tablet ddbc235ec29d41d28af358b96099f8d0 (DEFAULT_TABLE table=default.Table_53d2cf225a36435fbf6c220b550f8bba [id=ca35243dc04a461f90c257aa7ba27d2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.457990 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddbc235ec29d41d28af358b96099f8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.458299 4650 tablet_service.cc:1468] Processing CreateTablet for tablet ddbc235ec29d41d28af358b96099f8d0 (DEFAULT_TABLE table=default.Table_53d2cf225a36435fbf6c220b550f8bba [id=ca35243dc04a461f90c257aa7ba27d2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.459048 6000 raft_consensus.cc:491] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:46.459393 4650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddbc235ec29d41d28af358b96099f8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.459497 6000 raft_consensus.cc:513] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.461859 6000 leader_election.cc:290] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:46.463068 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d43617ed2cb4a0085e59eefcf8b19f1" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:46.465374 4755 leader_election.cc:343] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:46.465543 4942 tablet_service.cc:1468] Processing CreateTablet for tablet 134c7a645f5f4eb89b87929fb67dac97 (DEFAULT_TABLE table=default.Table_12a980fb917e4db18bfc5eeacda0d9d1 [id=ddec1a98a1434d7193e4e4fb63008ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.466223 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d43617ed2cb4a0085e59eefcf8b19f1" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:46.467682 4756 leader_election.cc:343] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:46.468052 4756 leader_election.cc:304] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:46.472805 4657 tablet_service.cc:1468] Processing CreateTablet for tablet 134c7a645f5f4eb89b87929fb67dac97 (DEFAULT_TABLE table=default.Table_12a980fb917e4db18bfc5eeacda0d9d1 [id=ddec1a98a1434d7193e4e4fb63008ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.473743 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 134c7a645f5f4eb89b87929fb67dac97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.473870 4942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 134c7a645f5f4eb89b87929fb67dac97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.475349 4649 tablet_service.cc:1468] Processing CreateTablet for tablet 9e4673214ad84f75a861c3af82a13d8f (DEFAULT_TABLE table=default.Table_fbcc45f0e38f4ed184900c0f58067165 [id=10f3db9788ba4edda2aab300e7313abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.476251 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e4673214ad84f75a861c3af82a13d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.476588 5353 ts_tablet_manager.cc:1428] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.072s user 0.005s sys 0.002s
I20250626 19:57:46.477423 5353 tablet_bootstrap.cc:492] T 125f7265258e48368381aa54559eb510 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:46.480890 4958 tablet_service.cc:1468] Processing CreateTablet for tablet 9e4673214ad84f75a861c3af82a13d8f (DEFAULT_TABLE table=default.Table_fbcc45f0e38f4ed184900c0f58067165 [id=10f3db9788ba4edda2aab300e7313abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.482132 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e4673214ad84f75a861c3af82a13d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.486217 6000 raft_consensus.cc:2747] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:46.493852 5907 log.cc:927] Time spent T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Append to log took a long time: real 0.051s user 0.004s sys 0.000s
I20250626 19:57:46.503041 4794 tablet_service.cc:1468] Processing CreateTablet for tablet 8c4549e621504a30ba5be5b0f139c905 (DEFAULT_TABLE table=default.Table_54b68b0dd5ba469cba186a5d4c6a9941 [id=7f7b8aca635746e0868116318bd54c0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.504575 4950 tablet_service.cc:1468] Processing CreateTablet for tablet f83ef2b2b82b4fa4996b12ce3d8b5401 (DEFAULT_TABLE table=default.Table_1bc851fd108147e3ab051386bab41342 [id=ceeb630cf3d842228d8319a65b2be238]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.505425 4794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c4549e621504a30ba5be5b0f139c905. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.505517 4950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f83ef2b2b82b4fa4996b12ce3d8b5401. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.509227 4793 tablet_service.cc:1468] Processing CreateTablet for tablet ddbc235ec29d41d28af358b96099f8d0 (DEFAULT_TABLE table=default.Table_53d2cf225a36435fbf6c220b550f8bba [id=ca35243dc04a461f90c257aa7ba27d2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.510246 4793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddbc235ec29d41d28af358b96099f8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.529260 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 134c7a645f5f4eb89b87929fb67dac97 (DEFAULT_TABLE table=default.Table_12a980fb917e4db18bfc5eeacda0d9d1 [id=ddec1a98a1434d7193e4e4fb63008ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.530314 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 134c7a645f5f4eb89b87929fb67dac97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.530567 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 9e4673214ad84f75a861c3af82a13d8f (DEFAULT_TABLE table=default.Table_fbcc45f0e38f4ed184900c0f58067165 [id=10f3db9788ba4edda2aab300e7313abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.531580 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e4673214ad84f75a861c3af82a13d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.264448 4609 leader_election.cc:304] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:46.565027 6015 raft_consensus.cc:2802] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:46.565541 6015 raft_consensus.cc:491] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:46.565964 6015 raft_consensus.cc:3058] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:46.576203 4797 tablet_service.cc:1468] Processing CreateTablet for tablet f83ef2b2b82b4fa4996b12ce3d8b5401 (DEFAULT_TABLE table=default.Table_1bc851fd108147e3ab051386bab41342 [id=ceeb630cf3d842228d8319a65b2be238]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.577183 4797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f83ef2b2b82b4fa4996b12ce3d8b5401. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.456123 5366 ts_tablet_manager.cc:1428] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.058s user 0.005s sys 0.000s
I20250626 19:57:46.582796 5366 tablet_bootstrap.cc:492] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:46.588111 5965 ts_tablet_manager.cc:1905] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:46.602769 4956 tablet_service.cc:1468] Processing CreateTablet for tablet 528e50de48f9459ebce01a00ce43d24b (DEFAULT_TABLE table=default.Table_3988af86d47e4eaa9103970949e16f12 [id=cd88a22d51a346deaf68390962fdb78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.603811 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 528e50de48f9459ebce01a00ce43d24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.613701 4648 tablet_service.cc:1468] Processing CreateTablet for tablet f83ef2b2b82b4fa4996b12ce3d8b5401 (DEFAULT_TABLE table=default.Table_1bc851fd108147e3ab051386bab41342 [id=ceeb630cf3d842228d8319a65b2be238]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.615134 4648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f83ef2b2b82b4fa4996b12ce3d8b5401. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:46.626915 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 365096us
I20250626 19:57:46.629593 4651 tablet_service.cc:1468] Processing CreateTablet for tablet 528e50de48f9459ebce01a00ce43d24b (DEFAULT_TABLE table=default.Table_3988af86d47e4eaa9103970949e16f12 [id=cd88a22d51a346deaf68390962fdb78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.635648 4653 tablet_service.cc:1468] Processing CreateTablet for tablet a8ef6f024d0642d4a6dd67d646787868 (DEFAULT_TABLE table=default.Table_d2c793438d234509974deb53c4e61c8b [id=84b3a43b3e5b42849f8b343a97906309]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.671429 4655 tablet_service.cc:1468] Processing CreateTablet for tablet 9a6d6380028445bf90931d334eeb5fbb (DEFAULT_TABLE table=default.Table_1fc5753cbe224eb4b476617a5d5d0d31 [id=971a1fb36a9248988a96ae65d9c80c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.671340 6015 raft_consensus.cc:513] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.680226 4959 tablet_service.cc:1468] Processing CreateTablet for tablet a8ef6f024d0642d4a6dd67d646787868 (DEFAULT_TABLE table=default.Table_d2c793438d234509974deb53c4e61c8b [id=84b3a43b3e5b42849f8b343a97906309]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.680827 5353 tablet_bootstrap.cc:700] T 125f7265258e48368381aa54559eb510 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent opening tablet: real 0.123s user 0.003s sys 0.000s
I20250626 19:57:46.681533 5353 tablet_bootstrap.cc:654] T 125f7265258e48368381aa54559eb510 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:46.682503 4953 tablet_service.cc:1468] Processing CreateTablet for tablet 9a6d6380028445bf90931d334eeb5fbb (DEFAULT_TABLE table=default.Table_1fc5753cbe224eb4b476617a5d5d0d31 [id=971a1fb36a9248988a96ae65d9c80c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.695415 4953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a6d6380028445bf90931d334eeb5fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.699921 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9172cb6576b240f6b25c3b1267618e6c" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:46.724736 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8be634cb36d40dca3f9338dab93ad2c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:46.728492 4912 leader_election.cc:304] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:46.733371 6026 raft_consensus.cc:3058] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:46.737672 5936 ts_tablet_manager.cc:1918] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:46.738101 5936 log.cc:1199] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/e94a73b50b564a97bbee0f5015bb8805
W20250626 19:57:46.738969 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 137158us
I20250626 19:57:46.706399 6015 leader_election.cc:290] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:46.757321 4651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 528e50de48f9459ebce01a00ce43d24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.758023 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ef6f024d0642d4a6dd67d646787868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.761691 4655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a6d6380028445bf90931d334eeb5fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.768976 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ef6f024d0642d4a6dd67d646787868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.757887 5936 ts_tablet_manager.cc:1939] T e94a73b50b564a97bbee0f5015bb8805 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:46.774931 4796 tablet_service.cc:1468] Processing CreateTablet for tablet 528e50de48f9459ebce01a00ce43d24b (DEFAULT_TABLE table=default.Table_3988af86d47e4eaa9103970949e16f12 [id=cd88a22d51a346deaf68390962fdb78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.790803 4796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 528e50de48f9459ebce01a00ce43d24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.793919 5936 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 158) took 1764 ms. Trace:
I20250626 19:57:46.794229 5936 rpcz_store.cc:276] 0626 19:57:45.029608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.029724 (+ 116us) service_pool.cc:225] Handling call
0626 19:57:46.332010 (+1302286us) ts_tablet_manager.cc:1145] Deleting tablet e94a73b50b564a97bbee0f5015bb8805
0626 19:57:46.332060 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:46.381261 (+ 49201us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:46.737422 (+356161us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:46.793859 (+ 56437us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1301690}
I20250626 19:57:46.807469 5936 tablet_replica.cc:331] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:46.809549 5410 tablet_bootstrap.cc:492] T 125f7265258e48368381aa54559eb510 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:46.809963 5410 ts_tablet_manager.cc:1397] T 125f7265258e48368381aa54559eb510 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.711s user 0.007s sys 0.006s
I20250626 19:57:46.812428 5937 ts_tablet_manager.cc:1918] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:57:46.814164 5937 log.cc:1199] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/3cc96a3e9c174cbca0196c5a1b27455c
I20250626 19:57:46.814870 5410 raft_consensus.cc:357] T 125f7265258e48368381aa54559eb510 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.815582 5410 raft_consensus.cc:383] T 125f7265258e48368381aa54559eb510 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:46.815848 5410 raft_consensus.cc:738] T 125f7265258e48368381aa54559eb510 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:46.816488 5410 consensus_queue.cc:260] T 125f7265258e48368381aa54559eb510 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.820005 5936 raft_consensus.cc:2241] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:46.821681 5936 raft_consensus.cc:2270] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:46.825985 5936 ts_tablet_manager.cc:1905] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:46.830410 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 67842.1us
I20250626 19:57:46.831712 5937 ts_tablet_manager.cc:1939] T 3cc96a3e9c174cbca0196c5a1b27455c P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:46.843197 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:46.852654 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 3cc96a3e9c174cbca0196c5a1b27455c (table default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]) successfully deleted
I20250626 19:57:46.861974 5965 ts_tablet_manager.cc:1918] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:57:46.862387 5965 log.cc:1199] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/3cc96a3e9c174cbca0196c5a1b27455c
I20250626 19:57:46.868131 5353 tablet_bootstrap.cc:492] T 125f7265258e48368381aa54559eb510 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:46.868583 5353 ts_tablet_manager.cc:1397] T 125f7265258e48368381aa54559eb510 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.391s user 0.014s sys 0.000s
W20250626 19:57:46.873240 4611 leader_election.cc:343] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:46.877503 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8be634cb36d40dca3f9338dab93ad2c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:46.877776 5366 tablet_bootstrap.cc:654] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:46.878059 4817 raft_consensus.cc:3058] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:46.844264 6026 raft_consensus.cc:2747] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:46.889644 5410 ts_tablet_manager.cc:1428] T 125f7265258e48368381aa54559eb510 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.079s user 0.007s sys 0.000s
I20250626 19:57:46.890630 5410 tablet_bootstrap.cc:492] T e9f09f0381eb420ca57c18a5be6f2cc2 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:46.896600 5965 ts_tablet_manager.cc:1939] T 3cc96a3e9c174cbca0196c5a1b27455c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:46.899948 5937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 175) took 1062 ms. Trace:
I20250626 19:57:46.900279 5937 rpcz_store.cc:276] 0626 19:57:45.837246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.837802 (+ 556us) service_pool.cc:225] Handling call
0626 19:57:46.899564 (+1061762us) ts_tablet_manager.cc:1145] Deleting tablet 1feb6785d0c7459fbc6898b0c9d3e38c
0626 19:57:46.899617 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:46.899878 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061252}
I20250626 19:57:46.901301 5937 tablet_replica.cc:331] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
W20250626 19:57:46.901890 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 1feb6785d0c7459fbc6898b0c9d3e38c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1feb6785d0c7459fbc6898b0c9d3e38c already in progress: creating tablet
I20250626 19:57:46.902020 5937 raft_consensus.cc:2241] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:46.917487 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet e94a73b50b564a97bbee0f5015bb8805 (table default.Table_fbcc45f0e38f4ed184900c0f58067165 [id=10f3db9788ba4edda2aab300e7313abb]) successfully deleted
I20250626 19:57:46.922854 4795 tablet_service.cc:1468] Processing CreateTablet for tablet 9a6d6380028445bf90931d334eeb5fbb (DEFAULT_TABLE table=default.Table_1fc5753cbe224eb4b476617a5d5d0d31 [id=971a1fb36a9248988a96ae65d9c80c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:46.927646 4392 master_service.cc:948] ReplaceTablet: received request to replace tablet 56f319c748374b7cbbf037ef55aecd1b from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:46.930586 5936 ts_tablet_manager.cc:1918] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:57:46.931015 5936 log.cc:1199] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/3cc96a3e9c174cbca0196c5a1b27455c
I20250626 19:57:46.932045 5936 ts_tablet_manager.cc:1939] T 3cc96a3e9c174cbca0196c5a1b27455c P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:46.949263 4817 raft_consensus.cc:2466] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:46.953557 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a6d6380028445bf90931d334eeb5fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:46.959282 4609 leader_election.cc:304] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:46.960007 6046 raft_consensus.cc:2802] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:46.960520 6046 raft_consensus.cc:695] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:46.961344 6046 consensus_queue.cc:237] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.902595 5937 pending_rounds.cc:70] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86: Trying to abort 1 pending ops.
I20250626 19:57:46.967374 4793 tablet_service.cc:1468] Processing CreateTablet for tablet a8ef6f024d0642d4a6dd67d646787868 (DEFAULT_TABLE table=default.Table_d2c793438d234509974deb53c4e61c8b [id=84b3a43b3e5b42849f8b343a97906309]), partition=RANGE (key) PARTITION UNBOUNDED
W20250626 19:57:46.968006 4384 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e9f09f0381eb420ca57c18a5be6f2cc2 consensus state has no leader
I20250626 19:57:46.966789 5937 pending_rounds.cc:77] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964378325643264 op_type: NO_OP noop_request { }
I20250626 19:57:46.971050 5937 raft_consensus.cc:2887] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:57:46.971987 5937 raft_consensus.cc:2270] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:46.972810 4392 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 56f319c748374b7cbbf037ef55aecd1b
I20250626 19:57:46.974308 4956 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f319c748374b7cbbf037ef55aecd1b with delete_type TABLET_DATA_DELETED (replaced by tablet 15aade4064d742488ce29145185551a7) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:46.974622 4392 catalog_manager.cc:6652] ReplaceTablet: tablet 56f319c748374b7cbbf037ef55aecd1b deleted and replaced by tablet 15aade4064d742488ce29145185551a7
I20250626 19:57:46.975806 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f319c748374b7cbbf037ef55aecd1b with delete_type TABLET_DATA_DELETED (replaced by tablet 15aade4064d742488ce29145185551a7) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:46.975934 4392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_1e3c7d9cf5334791b285859e997796ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:46.979112 5353 raft_consensus.cc:357] T 125f7265258e48368381aa54559eb510 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.979717 5353 raft_consensus.cc:383] T 125f7265258e48368381aa54559eb510 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:46.979951 5353 raft_consensus.cc:738] T 125f7265258e48368381aa54559eb510 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:46.984654 4796 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f319c748374b7cbbf037ef55aecd1b with delete_type TABLET_DATA_DELETED (replaced by tablet 15aade4064d742488ce29145185551a7) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:46.983494 5353 consensus_queue.cc:260] T 125f7265258e48368381aa54559eb510 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:46.985718 4793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ef6f024d0642d4a6dd67d646787868. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:46.999008 4392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e3c7d9cf5334791b285859e997796ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:46.940469 5965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 182) took 1185 ms. Trace:
I20250626 19:57:47.001971 6046 consensus_queue.cc:1035] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:47.011808 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:47.020983 5965 rpcz_store.cc:276] 0626 19:57:45.754229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.755926 (+ 1697us) service_pool.cc:225] Handling call
0626 19:57:46.376664 (+620738us) ts_tablet_manager.cc:1145] Deleting tablet 3cc96a3e9c174cbca0196c5a1b27455c
0626 19:57:46.376701 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:46.710313 (+333612us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:46.861748 (+151435us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:46.939897 (+ 78149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":620229}
I20250626 19:57:47.023769 5887 consensus_queue.cc:1035] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:47.044730 5937 ts_tablet_manager.cc:1905] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:47.047869 4392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_e2b576d684c543daaa81ab824be6ba7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:47.031564 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 95940.6us
I20250626 19:57:47.018488 4395 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 7851e464fcd44ecb82404d6a991db2a2 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]). Expected version 1 got 0
I20250626 19:57:47.049618 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 3cc96a3e9c174cbca0196c5a1b27455c (table default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]) successfully deleted
I20250626 19:57:47.049999 4395 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:47.051131 5965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 185) took 1256 ms. Trace:
I20250626 19:57:47.051468 5965 rpcz_store.cc:276] 0626 19:57:45.794446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.794581 (+ 135us) service_pool.cc:225] Handling call
0626 19:57:47.050695 (+1256114us) ts_tablet_manager.cc:1145] Deleting tablet fc526a05990446a3b3bc973e79511476
0626 19:57:47.050767 (+ 72us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.051068 (+ 301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1255443}
I20250626 19:57:47.039914 5353 ts_tablet_manager.cc:1428] T 125f7265258e48368381aa54559eb510 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.171s user 0.006s sys 0.003s
I20250626 19:57:47.059690 5353 tablet_bootstrap.cc:492] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:47.065045 4395 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:47.065723 4395 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:47.067008 4395 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 1474bfa74a6346bf9e3ff49019f6c9b3 (table default.Table_5fe142a9bd464f8ea2a070dda8bb61e9 [id=ab0bf254569748ba8d31931d4bfb105d]). Expected version 1 got 0
I20250626 19:57:47.067750 4395 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:47.067828 5936 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 163) took 1314 ms. Trace:
I20250626 19:57:47.068261 5936 rpcz_store.cc:276] 0626 19:57:45.753477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.753597 (+ 120us) service_pool.cc:225] Handling call
0626 19:57:46.807308 (+1053711us) ts_tablet_manager.cc:1145] Deleting tablet 3cc96a3e9c174cbca0196c5a1b27455c
0626 19:57:46.807347 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:46.900206 (+ 92859us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:46.930355 (+ 30149us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:47.067766 (+137411us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8553,"tablet-delete.queue_time_us":1051502}
I20250626 19:57:47.069707 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 3cc96a3e9c174cbca0196c5a1b27455c (table default.Table_d8774697646e49e9a77cfb883bae0f98 [id=b4d6a5e1732f4be39c13150d51816bb9]) successfully deleted
I20250626 19:57:47.070192 4395 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
W20250626 19:57:47.074903 4392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2b576d684c543daaa81ab824be6ba7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:47.095594 4395 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac]). Expected version 1 got 0
I20250626 19:57:47.098022 5366 tablet_bootstrap.cc:492] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:47.098409 5366 ts_tablet_manager.cc:1397] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.516s user 0.011s sys 0.001s
I20250626 19:57:47.108705 6044 raft_consensus.cc:491] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:47.109189 6044 raft_consensus.cc:513] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.111057 6044 leader_election.cc:290] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:47.111449 4395 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (table default.Table_8b997467fa764e8fbf273c3b1f2ccb3e [id=760c56838ee140d78aa368a3810d19a8]). Expected version 1 got 0
W20250626 19:57:47.113859 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet fc526a05990446a3b3bc973e79511476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc526a05990446a3b3bc973e79511476 already in progress: creating tablet
I20250626 19:57:47.113925 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc526a05990446a3b3bc973e79511476" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:47.114986 4756 leader_election.cc:343] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:47.115033 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc526a05990446a3b3bc973e79511476" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:47.115569 5907 log.cc:927] Time spent T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Append to log took a long time: real 0.057s user 0.003s sys 0.000s
W20250626 19:57:47.116286 4755 leader_election.cc:343] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:47.116662 4755 leader_election.cc:304] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:47.117733 6044 raft_consensus.cc:2747] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:47.151554 5410 tablet_bootstrap.cc:654] T e9f09f0381eb420ca57c18a5be6f2cc2 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:47.154160 5353 tablet_bootstrap.cc:654] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:47.155159 5965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 184) took 1361 ms. Trace:
I20250626 19:57:47.174700 5965 rpcz_store.cc:276] 0626 19:57:45.793852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.795445 (+ 1593us) service_pool.cc:225] Handling call
0626 19:57:47.110878 (+1315433us) ts_tablet_manager.cc:1145] Deleting tablet 6b66cda4f8ed410184b434d5083deac4
0626 19:57:47.154871 (+ 43993us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.155093 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1314917}
W20250626 19:57:47.175974 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 6b66cda4f8ed410184b434d5083deac4 because the tablet was not found. No further retry: Not found: Tablet not found: 6b66cda4f8ed410184b434d5083deac4
I20250626 19:57:47.177879 5366 raft_consensus.cc:357] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.197189 5936 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 164) took 1441 ms. Trace:
I20250626 19:57:47.197731 5936 rpcz_store.cc:276] 0626 19:57:45.755938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.756522 (+ 584us) service_pool.cc:225] Handling call
0626 19:57:47.147248 (+1390726us) ts_tablet_manager.cc:1145] Deleting tablet 1feb6785d0c7459fbc6898b0c9d3e38c
0626 19:57:47.186874 (+ 39626us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.197130 (+ 10256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1390210}
W20250626 19:57:47.199940 4383 catalog_manager.cc:4908] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 1feb6785d0c7459fbc6898b0c9d3e38c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1feb6785d0c7459fbc6898b0c9d3e38c already in progress: creating tablet
I20250626 19:57:47.192019 4870 heartbeater.cc:344] Connected to a master server at 127.4.46.190:33823
I20250626 19:57:47.224171 5965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 186) took 1426 ms. Trace:
I20250626 19:57:47.224874 6048 raft_consensus.cc:491] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:47.225306 6048 raft_consensus.cc:513] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.226709 4393 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:47.227679 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9f09f0381eb420ca57c18a5be6f2cc2" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:47.228864 6048 leader_election.cc:290] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:47.229161 4912 leader_election.cc:343] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:47.229524 4817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9f09f0381eb420ca57c18a5be6f2cc2" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:47.230029 4817 raft_consensus.cc:2466] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:47.230589 5410 tablet_bootstrap.cc:492] T e9f09f0381eb420ca57c18a5be6f2cc2 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:47.231009 5410 ts_tablet_manager.cc:1397] T e9f09f0381eb420ca57c18a5be6f2cc2 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.341s user 0.007s sys 0.003s
I20250626 19:57:47.231027 4911 leader_election.cc:304] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:47.231925 6048 raft_consensus.cc:2802] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:47.232249 6048 raft_consensus.cc:491] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:47.232568 6048 raft_consensus.cc:3058] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:47.233383 4393 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 7851e464fcd44ecb82404d6a991db2a2 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]). Expected version 1 got 0
I20250626 19:57:47.233258 5410 raft_consensus.cc:357] T e9f09f0381eb420ca57c18a5be6f2cc2 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.233963 5410 raft_consensus.cc:383] T e9f09f0381eb420ca57c18a5be6f2cc2 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:47.234225 5410 raft_consensus.cc:738] T e9f09f0381eb420ca57c18a5be6f2cc2 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:47.234886 5410 consensus_queue.cc:260] T e9f09f0381eb420ca57c18a5be6f2cc2 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.236406 4393 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:47.236974 5410 ts_tablet_manager.cc:1428] T e9f09f0381eb420ca57c18a5be6f2cc2 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250626 19:57:47.237815 5410 tablet_bootstrap.cc:492] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:47.178797 5366 raft_consensus.cc:383] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:47.239137 5366 raft_consensus.cc:738] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:47.240790 5366 consensus_queue.cc:260] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.239562 4393 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:47.244271 4393 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 1474bfa74a6346bf9e3ff49019f6c9b3 (table default.Table_5fe142a9bd464f8ea2a070dda8bb61e9 [id=ab0bf254569748ba8d31931d4bfb105d]). Expected version 1 got 0
I20250626 19:57:47.243989 5965 rpcz_store.cc:276] 0626 19:57:45.797360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.797502 (+ 142us) service_pool.cc:225] Handling call
0626 19:57:47.176984 (+1379482us) ts_tablet_manager.cc:1145] Deleting tablet cb34ffcd4da04e32a1220b3f298ec22d
0626 19:57:47.223923 (+ 46939us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.224117 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1378932}
I20250626 19:57:47.245352 5965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 188) took 1445 ms. Trace:
I20250626 19:57:47.245656 5965 rpcz_store.cc:276] 0626 19:57:45.800420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.800542 (+ 122us) service_pool.cc:225] Handling call
0626 19:57:47.244998 (+1444456us) ts_tablet_manager.cc:1145] Deleting tablet f807cf54bbb146ff989762c18410e6bb
0626 19:57:47.245040 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.245302 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1443254}
W20250626 19:57:47.245378 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet cb34ffcd4da04e32a1220b3f298ec22d because the tablet was not found. No further retry: Not found: Tablet not found: cb34ffcd4da04e32a1220b3f298ec22d
W20250626 19:57:47.251350 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet f807cf54bbb146ff989762c18410e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f807cf54bbb146ff989762c18410e6bb already in progress: creating tablet
I20250626 19:57:47.253355 5936 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 165) took 1301 ms. Trace:
I20250626 19:57:47.253690 5936 rpcz_store.cc:276] 0626 19:57:45.951872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.951980 (+ 108us) service_pool.cc:225] Handling call
0626 19:57:47.253028 (+1301048us) ts_tablet_manager.cc:1145] Deleting tablet 4c0338c05a974e0f96865398722c5e8f
0626 19:57:47.253073 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.253288 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1300459}
I20250626 19:57:47.251626 4393 catalog_manager.cc:5582] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
W20250626 19:57:47.254874 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 4c0338c05a974e0f96865398722c5e8f because the tablet was not found. No further retry: Not found: Tablet not found: 4c0338c05a974e0f96865398722c5e8f
I20250626 19:57:47.255676 4393 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:47.259739 5936 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 166) took 1303 ms. Trace:
I20250626 19:57:47.269207 4393 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac]). Expected version 1 got 0
I20250626 19:57:47.279547 4393 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:47.280268 4393 catalog_manager.cc:5582] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:47.285482 4393 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (table default.Table_8b997467fa764e8fbf273c3b1f2ccb3e [id=760c56838ee140d78aa368a3810d19a8]). Expected version 1 got 0
I20250626 19:57:47.292841 6048 raft_consensus.cc:513] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.300662 4820 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9f09f0381eb420ca57c18a5be6f2cc2" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:47.246606 5965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 187) took 1446 ms. Trace:
I20250626 19:57:47.301257 4820 raft_consensus.cc:3058] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:47.301481 5965 rpcz_store.cc:276] 0626 19:57:45.799861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.804293 (+ 4432us) service_pool.cc:225] Handling call
0626 19:57:47.246382 (+1442089us) ts_tablet_manager.cc:1145] Deleting tablet 609d36d46bb84017b2a3a7ad48bfccfc
0626 19:57:47.246418 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.246561 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1441160}
W20250626 19:57:47.303277 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 609d36d46bb84017b2a3a7ad48bfccfc because the tablet was not found. No further retry: Not found: Tablet not found: 609d36d46bb84017b2a3a7ad48bfccfc
I20250626 19:57:47.303444 5965 tablet_replica.cc:331] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:47.260552 5936 rpcz_store.cc:276] 0626 19:57:45.955993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.956122 (+ 129us) service_pool.cc:225] Handling call
0626 19:57:47.256583 (+1300461us) ts_tablet_manager.cc:1145] Deleting tablet f6ec952321924098a142f1438a283aab
0626 19:57:47.256714 (+ 131us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.259599 (+ 2885us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1299237}
W20250626 19:57:47.306558 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet f6ec952321924098a142f1438a283aab because the tablet was not found. No further retry: Not found: Tablet not found: f6ec952321924098a142f1438a283aab
I20250626 19:57:47.308018 5936 tablet_replica.cc:331] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:47.309754 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9f09f0381eb420ca57c18a5be6f2cc2" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:47.310374 4670 raft_consensus.cc:3058] T e9f09f0381eb420ca57c18a5be6f2cc2 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:47.314097 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:47.328094 5353 tablet_bootstrap.cc:492] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:47.328608 5353 ts_tablet_manager.cc:1397] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.269s user 0.002s sys 0.009s
I20250626 19:57:47.321527 4670 raft_consensus.cc:2466] T e9f09f0381eb420ca57c18a5be6f2cc2 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:47.329792 4912 leader_election.cc:304] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:47.330615 5924 raft_consensus.cc:2802] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:47.331045 5924 raft_consensus.cc:695] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:47.331780 5924 consensus_queue.cc:237] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.337606 5410 tablet_bootstrap.cc:654] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:47.348917 5366 ts_tablet_manager.cc:1428] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.250s user 0.007s sys 0.000s
I20250626 19:57:47.349020 4820 raft_consensus.cc:2466] T e9f09f0381eb420ca57c18a5be6f2cc2 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:47.354991 5366 tablet_bootstrap.cc:492] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:47.355674 6045 raft_consensus.cc:491] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:47.356078 6045 raft_consensus.cc:513] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.365067 6045 leader_election.cc:290] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:47.369783 5353 raft_consensus.cc:357] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.317548 6048 leader_election.cc:290] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:47.379241 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "125f7265258e48368381aa54559eb510" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:47.379961 4670 raft_consensus.cc:2466] T 125f7265258e48368381aa54559eb510 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:47.373899 5965 raft_consensus.cc:2241] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:47.416083 5965 raft_consensus.cc:2270] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:47.417903 5965 ts_tablet_manager.cc:1905] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:47.373634 5936 raft_consensus.cc:2241] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:47.380970 4755 leader_election.cc:304] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
W20250626 19:57:47.441828 4755 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 60967.9us
I20250626 19:57:47.442617 6044 raft_consensus.cc:2802] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:47.442966 6044 raft_consensus.cc:491] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:47.443275 6044 raft_consensus.cc:3058] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:47.414496 6045 raft_consensus.cc:491] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:47.435621 5936 raft_consensus.cc:2270] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:47.451342 6045 raft_consensus.cc:513] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.450675 6044 raft_consensus.cc:513] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.456365 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "125f7265258e48368381aa54559eb510" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:47.456904 4670 raft_consensus.cc:3058] T 125f7265258e48368381aa54559eb510 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:47.462858 6044 leader_election.cc:290] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:47.483448 5366 tablet_bootstrap.cc:654] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:47.483675 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 5fc2e6fb07b3407783861af7b5c2c99e (DEFAULT_TABLE table=default.Table_9357cbde46f74ae583f88b25120e4811 [id=e3f36704afb446f2a32095326c0cccd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.487459 4653 tablet_service.cc:1468] Processing CreateTablet for tablet 5fc2e6fb07b3407783861af7b5c2c99e (DEFAULT_TABLE table=default.Table_9357cbde46f74ae583f88b25120e4811 [id=e3f36704afb446f2a32095326c0cccd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.488492 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fc2e6fb07b3407783861af7b5c2c99e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.488859 4793 tablet_service.cc:1468] Processing CreateTablet for tablet 5fc2e6fb07b3407783861af7b5c2c99e (DEFAULT_TABLE table=default.Table_9357cbde46f74ae583f88b25120e4811 [id=e3f36704afb446f2a32095326c0cccd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.489792 4793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fc2e6fb07b3407783861af7b5c2c99e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.503687 4953 tablet_service.cc:1468] Processing CreateTablet for tablet d8048a3aea23413c8f2cd07a69fd73fa (DEFAULT_TABLE table=default.Table_0bdf32dfb89d4500877aeed0695d926f [id=886a810aba7d4918b680b3720d846b6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.505798 4649 tablet_service.cc:1468] Processing CreateTablet for tablet d8048a3aea23413c8f2cd07a69fd73fa (DEFAULT_TABLE table=default.Table_0bdf32dfb89d4500877aeed0695d926f [id=886a810aba7d4918b680b3720d846b6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.506742 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8048a3aea23413c8f2cd07a69fd73fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.507275 4795 tablet_service.cc:1468] Processing CreateTablet for tablet d8048a3aea23413c8f2cd07a69fd73fa (DEFAULT_TABLE table=default.Table_0bdf32dfb89d4500877aeed0695d926f [id=886a810aba7d4918b680b3720d846b6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.508658 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8048a3aea23413c8f2cd07a69fd73fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.519722 4670 raft_consensus.cc:2466] T 125f7265258e48368381aa54559eb510 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:47.521413 4755 leader_election.cc:304] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:47.526044 6045 leader_election.cc:290] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:47.529459 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:47.542053 5937 ts_tablet_manager.cc:1918] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:47.542445 5937 log.cc:1199] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/9172cb6576b240f6b25c3b1267618e6c
I20250626 19:57:47.544553 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 7851e464fcd44ecb82404d6a991db2a2 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]). Expected version 1 got 0
I20250626 19:57:47.533272 6045 raft_consensus.cc:2802] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:47.549486 6045 raft_consensus.cc:695] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:47.549844 4956 tablet_service.cc:1468] Processing CreateTablet for tablet 3153fd2245de49b29eabd1b7fcc24e35 (DEFAULT_TABLE table=default.Table_3bffc9f08d164dafb33290412ccbd109 [id=fece4e537afc4e209c1ea5375dd40739]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.556049 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f807cf54bbb146ff989762c18410e6bb" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:47.560791 4655 tablet_service.cc:1468] Processing CreateTablet for tablet 3153fd2245de49b29eabd1b7fcc24e35 (DEFAULT_TABLE table=default.Table_3bffc9f08d164dafb33290412ccbd109 [id=fece4e537afc4e209c1ea5375dd40739]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.561811 4655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3153fd2245de49b29eabd1b7fcc24e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.547349 5936 ts_tablet_manager.cc:1905] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:47.558329 4755 leader_election.cc:343] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:47.370544 5353 raft_consensus.cc:383] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:47.563520 5353 raft_consensus.cc:738] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:47.564402 5353 consensus_queue.cc:260] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.568260 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "125f7265258e48368381aa54559eb510" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:47.568830 4976 raft_consensus.cc:2466] T 125f7265258e48368381aa54559eb510 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:47.572521 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "125f7265258e48368381aa54559eb510" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:47.573050 4976 raft_consensus.cc:3058] T 125f7265258e48368381aa54559eb510 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:47.571810 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:47.575134 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fc2e6fb07b3407783861af7b5c2c99e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.575791 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3153fd2245de49b29eabd1b7fcc24e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.576355 4953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8048a3aea23413c8f2cd07a69fd73fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.577351 5410 tablet_bootstrap.cc:492] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:47.577590 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:47.578056 5410 ts_tablet_manager.cc:1397] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.340s user 0.009s sys 0.011s
I20250626 19:57:47.562248 6045 consensus_queue.cc:237] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.583338 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f807cf54bbb146ff989762c18410e6bb" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:47.595697 4796 tablet_service.cc:1468] Processing CreateTablet for tablet 3153fd2245de49b29eabd1b7fcc24e35 (DEFAULT_TABLE table=default.Table_3bffc9f08d164dafb33290412ccbd109 [id=fece4e537afc4e209c1ea5375dd40739]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.596446 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (table default.Table_56852b6e38af4545879ff672cc528975 [id=e3400f897e9944549162854e3f92ac21]). Expected version 1 got 0
I20250626 19:57:47.596658 4796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3153fd2245de49b29eabd1b7fcc24e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.596993 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 1474bfa74a6346bf9e3ff49019f6c9b3 (table default.Table_5fe142a9bd464f8ea2a070dda8bb61e9 [id=ab0bf254569748ba8d31931d4bfb105d]). Expected version 1 got 0
I20250626 19:57:47.597674 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:47.608915 5366 tablet_bootstrap.cc:492] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:47.609421 5366 ts_tablet_manager.cc:1397] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.255s user 0.012s sys 0.003s
I20250626 19:57:47.598740 4391 catalog_manager.cc:5582] T ef1abf340cc34e00aa67e4588c61c32d P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:47.611590 4976 raft_consensus.cc:2466] T 125f7265258e48368381aa54559eb510 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:47.612926 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:47.616187 4653 tablet_service.cc:1468] Processing CreateTablet for tablet 3c70eccce0ce485e8d1679b103e130dc (DEFAULT_TABLE table=default.Table_d849d42d83a746bcabb98ca420c0ec24 [id=377f58d807124b11b1d029bce34c12fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.617225 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c70eccce0ce485e8d1679b103e130dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.620021 4657 tablet_service.cc:1468] Processing CreateTablet for tablet f0d01ad0685442748d534c5113610bf8 (DEFAULT_TABLE table=default.Table_ec900845fd1c45e68e99858c077c57ac [id=2467561841fe414199e2c486635fea55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.621081 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d01ad0685442748d534c5113610bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.619776 4956 tablet_service.cc:1468] Processing CreateTablet for tablet 3c70eccce0ce485e8d1679b103e130dc (DEFAULT_TABLE table=default.Table_d849d42d83a746bcabb98ca420c0ec24 [id=377f58d807124b11b1d029bce34c12fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.624505 4391 catalog_manager.cc:5582] T 69bd106066b6492bb7ea03d15a18df00 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:47.625530 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c70eccce0ce485e8d1679b103e130dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.631759 4794 tablet_service.cc:1468] Processing CreateTablet for tablet 3c70eccce0ce485e8d1679b103e130dc (DEFAULT_TABLE table=default.Table_d849d42d83a746bcabb98ca420c0ec24 [id=377f58d807124b11b1d029bce34c12fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.632866 4794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c70eccce0ce485e8d1679b103e130dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.632647 5366 raft_consensus.cc:357] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.633256 5366 raft_consensus.cc:383] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:47.633491 5366 raft_consensus.cc:738] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:47.634001 4953 tablet_service.cc:1468] Processing CreateTablet for tablet f0d01ad0685442748d534c5113610bf8 (DEFAULT_TABLE table=default.Table_ec900845fd1c45e68e99858c077c57ac [id=2467561841fe414199e2c486635fea55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.634060 5366 consensus_queue.cc:260] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.635106 4953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d01ad0685442748d534c5113610bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.639456 5937 ts_tablet_manager.cc:1939] T 9172cb6576b240f6b25c3b1267618e6c P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:47.639588 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac]). Expected version 1 got 0
I20250626 19:57:47.640242 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (table default.Table_8b997467fa764e8fbf273c3b1f2ccb3e [id=760c56838ee140d78aa368a3810d19a8]). Expected version 1 got 0
I20250626 19:57:47.640285 5937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 177) took 1566 ms. Trace:
I20250626 19:57:47.640583 5937 rpcz_store.cc:276] 0626 19:57:46.073733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.073830 (+ 97us) service_pool.cc:225] Handling call
0626 19:57:46.901150 (+827320us) ts_tablet_manager.cc:1145] Deleting tablet 9172cb6576b240f6b25c3b1267618e6c
0626 19:57:46.901184 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.281309 (+380125us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:47.541820 (+260511us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:47.640230 (+ 98410us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":826829}
I20250626 19:57:47.641733 5937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 178) took 1442 ms. Trace:
I20250626 19:57:47.641708 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 9172cb6576b240f6b25c3b1267618e6c (table default.Table_2e7897256c7140aeb545751922965c14 [id=441b14db312f4da1be3f2219cf745bc6]) successfully deleted
I20250626 19:57:47.642019 5937 rpcz_store.cc:276] 0626 19:57:46.198917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.199023 (+ 106us) service_pool.cc:225] Handling call
0626 19:57:47.641503 (+1442480us) ts_tablet_manager.cc:1145] Deleting tablet 4c0338c05a974e0f96865398722c5e8f
0626 19:57:47.641540 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.641683 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1441943}
W20250626 19:57:47.642993 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 4c0338c05a974e0f96865398722c5e8f because the tablet was not found. No further retry: Not found: Tablet not found: 4c0338c05a974e0f96865398722c5e8f
I20250626 19:57:47.643079 4391 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 34b2919e5f534f4295336a4384be1f64 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:47.658263 5907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Bootstrap replayed 1/1 log segments. Stats: ops{read=87 overwritten=0 applied=84 ignored=0} inserts{seen=75 ignored=0} mutations{seen=72 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250626 19:57:47.659374 5907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Bootstrap complete.
I20250626 19:57:47.668921 5410 raft_consensus.cc:357] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.669750 5410 raft_consensus.cc:383] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:47.670058 5410 raft_consensus.cc:738] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:47.678407 4655 tablet_service.cc:1468] Processing CreateTablet for tablet cbbca3880de54980a3673a281509ff3c (DEFAULT_TABLE table=default.Table_3f76c8e8c67b452cb9dd1c2817f493b8 [id=6549866f84b24fb396f4c75861f802d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.679502 4655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbbca3880de54980a3673a281509ff3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.684132 5410 consensus_queue.cc:260] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.688385 4795 tablet_service.cc:1468] Processing CreateTablet for tablet f0d01ad0685442748d534c5113610bf8 (DEFAULT_TABLE table=default.Table_ec900845fd1c45e68e99858c077c57ac [id=2467561841fe414199e2c486635fea55]), partition=RANGE (key) PARTITION UNBOUNDED
W20250626 19:57:47.625644 4756 leader_election.cc:343] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:47.695206 4756 leader_election.cc:304] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
W20250626 19:57:47.696031 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 70591.9us
I20250626 19:57:47.699970 5410 ts_tablet_manager.cc:1428] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.122s user 0.003s sys 0.002s
I20250626 19:57:47.703271 5410 tablet_bootstrap.cc:492] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:47.719626 4950 tablet_service.cc:1468] Processing CreateTablet for tablet b1d2caf51dec4186a486fc806941ced3 (DEFAULT_TABLE table=default.Table_b574eeb305114df283c62fab66bdafb3 [id=00297b7294ec478f86fc6e1989e22541]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.720633 4950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1d2caf51dec4186a486fc806941ced3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.721493 4958 tablet_service.cc:1468] Processing CreateTablet for tablet 8e3d836b65174c92b698ed77a6e87e49 (DEFAULT_TABLE table=default.Table_714f173170274ff7ac112d613b731467 [id=e441e02dd8fe4719be2208c52ab725ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.722465 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3d836b65174c92b698ed77a6e87e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.730489 4797 tablet_service.cc:1468] Processing CreateTablet for tablet cbbca3880de54980a3673a281509ff3c (DEFAULT_TABLE table=default.Table_3f76c8e8c67b452cb9dd1c2817f493b8 [id=6549866f84b24fb396f4c75861f802d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.704255 4657 tablet_service.cc:1468] Processing CreateTablet for tablet b1d2caf51dec4186a486fc806941ced3 (DEFAULT_TABLE table=default.Table_b574eeb305114df283c62fab66bdafb3 [id=00297b7294ec478f86fc6e1989e22541]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.710978 4649 tablet_service.cc:1468] Processing CreateTablet for tablet 8e3d836b65174c92b698ed77a6e87e49 (DEFAULT_TABLE table=default.Table_714f173170274ff7ac112d613b731467 [id=e441e02dd8fe4719be2208c52ab725ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.730857 4800 tablet_service.cc:1468] Processing CreateTablet for tablet b1d2caf51dec4186a486fc806941ced3 (DEFAULT_TABLE table=default.Table_b574eeb305114df283c62fab66bdafb3 [id=00297b7294ec478f86fc6e1989e22541]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.733163 4799 tablet_service.cc:1468] Processing CreateTablet for tablet 8e3d836b65174c92b698ed77a6e87e49 (DEFAULT_TABLE table=default.Table_714f173170274ff7ac112d613b731467 [id=e441e02dd8fe4719be2208c52ab725ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.739984 5937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 181) took 1501 ms. Trace:
I20250626 19:57:47.739717 4951 tablet_service.cc:1468] Processing CreateTablet for tablet cbbca3880de54980a3673a281509ff3c (DEFAULT_TABLE table=default.Table_3f76c8e8c67b452cb9dd1c2817f493b8 [id=6549866f84b24fb396f4c75861f802d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.740300 5937 rpcz_store.cc:276] 0626 19:57:46.237424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.237555 (+ 131us) service_pool.cc:225] Handling call
0626 19:57:47.703773 (+1466218us) ts_tablet_manager.cc:1145] Deleting tablet 36997dab0ab3430bba63a90805759b30
0626 19:57:47.738860 (+ 35087us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.739021 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1404603}
I20250626 19:57:47.740829 4951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbbca3880de54980a3673a281509ff3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.741400 5937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 182) took 1503 ms. Trace:
I20250626 19:57:47.741662 5937 rpcz_store.cc:276] 0626 19:57:46.237737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.238635 (+ 898us) service_pool.cc:225] Handling call
0626 19:57:47.741098 (+1502463us) ts_tablet_manager.cc:1145] Deleting tablet fc526a05990446a3b3bc973e79511476
0626 19:57:47.741135 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.741354 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1501971}
I20250626 19:57:47.742587 5937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 183) took 1504 ms. Trace:
W20250626 19:57:47.743152 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 36997dab0ab3430bba63a90805759b30 because the tablet was not found. No further retry: Not found: Tablet not found: 36997dab0ab3430bba63a90805759b30
W20250626 19:57:47.743839 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet fc526a05990446a3b3bc973e79511476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc526a05990446a3b3bc973e79511476 already in progress: creating tablet
I20250626 19:57:47.744406 5353 ts_tablet_manager.cc:1428] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.416s user 0.005s sys 0.001s
I20250626 19:57:47.743121 5937 rpcz_store.cc:276] 0626 19:57:46.237993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.239216 (+ 1223us) service_pool.cc:225] Handling call
0626 19:57:47.742379 (+1503163us) ts_tablet_manager.cc:1145] Deleting tablet cb34ffcd4da04e32a1220b3f298ec22d
0626 19:57:47.742413 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.742545 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1502626}
W20250626 19:57:47.745251 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet cb34ffcd4da04e32a1220b3f298ec22d because the tablet was not found. No further retry: Not found: Tablet not found: cb34ffcd4da04e32a1220b3f298ec22d
I20250626 19:57:47.746320 5937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 185) took 1507 ms. Trace:
I20250626 19:57:47.751411 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1d2caf51dec4186a486fc806941ced3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.752027 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3d836b65174c92b698ed77a6e87e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.763715 6066 raft_consensus.cc:491] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:47.764146 6066 raft_consensus.cc:513] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.765928 6066 leader_election.cc:290] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:47.767071 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c5aba053ca4535ad2376b2066cfe47" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:47.768682 4755 leader_election.cc:343] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:47.747117 5937 rpcz_store.cc:276] 0626 19:57:46.238529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.239963 (+ 1434us) service_pool.cc:225] Handling call
0626 19:57:47.745465 (+1505502us) ts_tablet_manager.cc:1145] Deleting tablet f807cf54bbb146ff989762c18410e6bb
0626 19:57:47.745589 (+ 124us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.746182 (+ 593us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1504951}
I20250626 19:57:47.731793 6044 raft_consensus.cc:2747] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:47.771221 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet f807cf54bbb146ff989762c18410e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f807cf54bbb146ff989762c18410e6bb already in progress: creating tablet
I20250626 19:57:47.781828 4797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbbca3880de54980a3673a281509ff3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.787204 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d01ad0685442748d534c5113610bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.787878 5366 ts_tablet_manager.cc:1428] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.178s user 0.008s sys 0.016s
I20250626 19:57:47.788645 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3d836b65174c92b698ed77a6e87e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.791157 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1d2caf51dec4186a486fc806941ced3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.791450 5937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 180) took 1553 ms. Trace:
I20250626 19:57:47.799328 5366 tablet_bootstrap.cc:492] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:47.797390 5937 rpcz_store.cc:276] 0626 19:57:46.237104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.240769 (+ 3665us) service_pool.cc:225] Handling call
0626 19:57:47.784178 (+1543409us) ts_tablet_manager.cc:1145] Deleting tablet f6ec952321924098a142f1438a283aab
0626 19:57:47.785674 (+ 1496us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.789630 (+ 3956us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542103}
I20250626 19:57:47.803646 5937 tablet_replica.cc:331] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
W20250626 19:57:47.803647 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet f6ec952321924098a142f1438a283aab because the tablet was not found. No further retry: Not found: Tablet not found: f6ec952321924098a142f1438a283aab
I20250626 19:57:47.804358 5937 raft_consensus.cc:2241] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:47.804769 5937 raft_consensus.cc:2270] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:47.806360 5937 ts_tablet_manager.cc:1905] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:47.793211 5907 raft_consensus.cc:357] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 5 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:47.821722 5366 tablet_bootstrap.cc:654] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:47.824235 4392 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 7851e464fcd44ecb82404d6a991db2a2 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]). Expected version 1 got 0
I20250626 19:57:47.826728 4392 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:47.831645 4649 tablet_service.cc:1468] Processing CreateTablet for tablet 290232b0933b451c9d7744165bd72796 (DEFAULT_TABLE table=default.Table_2e7897256c7140aeb545751922965c14 [id=441b14db312f4da1be3f2219cf745bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.832602 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290232b0933b451c9d7744165bd72796. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.840818 5353 tablet_bootstrap.cc:492] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:47.844960 4956 tablet_service.cc:1468] Processing CreateTablet for tablet 290232b0933b451c9d7744165bd72796 (DEFAULT_TABLE table=default.Table_2e7897256c7140aeb545751922965c14 [id=441b14db312f4da1be3f2219cf745bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.846154 4956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290232b0933b451c9d7744165bd72796. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.867399 5366 tablet_bootstrap.cc:492] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:47.873116 5907 raft_consensus.cc:738] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c58006b6be41c1bd6bdb5eb010d360, State: Initialized, Role: FOLLOWER
I20250626 19:57:47.874207 5366 ts_tablet_manager.cc:1397] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.005s
I20250626 19:57:47.876021 4392 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (table default.Table_56852b6e38af4545879ff672cc528975 [id=e3400f897e9944549162854e3f92ac21]). Expected version 1 got 0
I20250626 19:57:47.882253 6046 consensus_queue.cc:1035] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:47.891165 5936 ts_tablet_manager.cc:1918] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:47.891530 5936 log.cc:1199] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/fc526a05990446a3b3bc973e79511476
I20250626 19:57:47.892118 5936 ts_tablet_manager.cc:1939] T fc526a05990446a3b3bc973e79511476 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:47.892812 5936 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 167) took 1922 ms. Trace:
I20250626 19:57:47.893110 5936 rpcz_store.cc:276] 0626 19:57:45.970265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.970415 (+ 150us) service_pool.cc:225] Handling call
0626 19:57:47.307531 (+1337116us) ts_tablet_manager.cc:1145] Deleting tablet fc526a05990446a3b3bc973e79511476
0626 19:57:47.307673 (+ 142us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.729284 (+421611us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:47.890941 (+161657us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:47.892762 (+ 1821us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1336478}
I20250626 19:57:47.894181 5936 tablet_replica.cc:331] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:47.904990 5410 tablet_bootstrap.cc:654] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:47.912210 6069 raft_consensus.cc:491] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:47.902025 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet fc526a05990446a3b3bc973e79511476 (table default.Table_38eec4fd44e848378efabe2ebbc88983 [id=4e600070f9e44758a4b11f0adeab28e5]) successfully deleted
I20250626 19:57:47.913467 6069 raft_consensus.cc:513] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:47.915230 4392 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:47.915719 4794 tablet_service.cc:1468] Processing CreateTablet for tablet 290232b0933b451c9d7744165bd72796 (DEFAULT_TABLE table=default.Table_2e7897256c7140aeb545751922965c14 [id=441b14db312f4da1be3f2219cf745bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.919567 4392 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:47.919868 4796 tablet_service.cc:1468] Processing CreateTablet for tablet cdb9c1a7f8d64821b2efc4cdb655382b (DEFAULT_TABLE table=default.Table_65406953a5444c54a77eaed676d03f6c [id=819506362a8e4070b951855ee287fe33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.922541 4392 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:47.874205 5907 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 5.87, Last appended by leader: 87, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:47.932080 5965 ts_tablet_manager.cc:1918] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:47.932471 5965 log.cc:1199] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/9172cb6576b240f6b25c3b1267618e6c
I20250626 19:57:47.933121 5965 ts_tablet_manager.cc:1939] T 9172cb6576b240f6b25c3b1267618e6c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
W20250626 19:57:47.934885 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 219655us
I20250626 19:57:47.940606 4392 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:47.939253 5366 raft_consensus.cc:357] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:47.941160 5366 raft_consensus.cc:383] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:47.941496 5366 raft_consensus.cc:738] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:47.942116 5366 consensus_queue.cc:260] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:47.945065 6069 leader_election.cc:290] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:47.940194 5936 raft_consensus.cc:2241] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:47.947701 4392 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac]). Expected version 1 got 0
I20250626 19:57:47.948863 4794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290232b0933b451c9d7744165bd72796. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.949841 4392 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (table default.Table_8b997467fa764e8fbf273c3b1f2ccb3e [id=760c56838ee140d78aa368a3810d19a8]). Expected version 1 got 0
I20250626 19:57:47.952167 4392 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 34b2919e5f534f4295336a4384be1f64 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:47.953552 4796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb9c1a7f8d64821b2efc4cdb655382b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:47.949878 5936 raft_consensus.cc:2270] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:47.961596 5366 ts_tablet_manager.cc:1428] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.069s user 0.005s sys 0.000s
I20250626 19:57:47.964558 4793 tablet_service.cc:1468] Processing CreateTablet for tablet 54271a1d675e41018c2f2b2a275d761a (DEFAULT_TABLE table=default.Table_90bb38ee5f4c4b3ca1d9f018e53ef6b9 [id=4e809bf42770487ea08e0cc70065a28b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:47.965481 4398 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 54e68eba747e4a2a947e84e53a444d08
I20250626 19:57:47.966862 4398 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34796 (request call id 174) took 2003 ms. Trace:
I20250626 19:57:47.967145 4398 rpcz_store.cc:276] 0626 19:57:45.963090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.071789 (+108699us) service_pool.cc:225] Handling call
0626 19:57:46.072902 (+ 1113us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:46.113458 (+ 40556us) catalog_manager.cc:2790] Modifying in-memory table state
0626 19:57:46.113590 (+ 132us) catalog_manager.cc:2799] Locking tablets
0626 19:57:46.412275 (+298685us) catalog_manager.cc:2813] Removing table and tablets from system table
0626 19:57:46.657924 (+245649us) catalog_manager.cc:2831] Removing table from by-name map
0626 19:57:46.917128 (+259204us) catalog_manager.cc:2847] Committing in-memory state
0626 19:57:47.965383 (+1048255us) catalog_manager.cc:2851] Aborting table tasks
0626 19:57:47.966802 (+ 1419us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:46.436765 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:46.437307 (+ 542us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:46.437340 (+ 33us) write_op.cc:435] Acquired schema lock
0626 19:57:46.437353 (+ 13us) tablet.cc:660] Decoding operations
0626 19:57:46.437743 (+ 390us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:46.437791 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:46.437828 (+ 37us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:46.438143 (+ 315us) tablet.cc:699] Row locks acquired
0626 19:57:46.438155 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:46.438277 (+ 122us) write_op.cc:270] Start()
0626 19:57:46.438365 (+ 88us) write_op.cc:276] Timestamp: P: 1750967866438414 usec, L: 0
0626 19:57:46.438384 (+ 19us) op_driver.cc:348] REPLICATION: starting
0626 19:57:46.438943 (+ 559us) log.cc:844] Serialized 728 byte log entry
0626 19:57:46.655985 (+217042us) op_driver.cc:464] REPLICATION: finished
0626 19:57:46.656302 (+ 317us) write_op.cc:301] APPLY: starting
0626 19:57:46.656357 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:46.656471 (+ 114us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:46.656557 (+ 86us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:46.656786 (+ 229us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:46.656938 (+ 152us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:46.656983 (+ 45us) write_op.cc:312] APPLY: finished
0626 19:57:46.657162 (+ 179us) log.cc:844] Serialized 29 byte log entry
0626 19:57:46.657310 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:46.657390 (+ 80us) write_op.cc:454] Released schema lock
0626 19:57:46.657420 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:46.657636 (+ 216us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:46.657665 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":169,"apply.run_cpu_time_us":1313,"apply.run_wall_time_us":1312,"num_ops":2,"prepare.queue_time_us":22833,"prepare.run_cpu_time_us":2315,"prepare.run_wall_time_us":2401,"replication_time_us":217498}]]}
I20250626 19:57:47.975334 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 54e68eba747e4a2a947e84e53a444d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:47.975603 6076 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:47.976466 6076 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:47.976758 4801 tablet_service.cc:1468] Processing CreateTablet for tablet a319f0cb00f243f08c1cb2f3efd94db1 (DEFAULT_TABLE table=default.Table_d9f785633e4f45eaa38fd0ec36f46224 [id=4796f278305d4f66a36d676736e72b23]), partition=RANGE (key) PARTITION UNBOUNDED
W20250626 19:57:47.983804 4396 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 27acc0eafbb04375ba1c6a59e958e6f6 consensus state has no leader
I20250626 19:57:47.983970 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83f0431638144710b598b3f87d9a576e" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:47.984577 5366 tablet_bootstrap.cc:492] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:47.984935 4793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54271a1d675e41018c2f2b2a275d761a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:57:47.986845 4756 leader_election.cc:343] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:47.993266 5965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 189) took 1919 ms. Trace:
I20250626 19:57:47.998168 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a319f0cb00f243f08c1cb2f3efd94db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:48.000907 4942 tablet_service.cc:1515] Processing DeleteTablet for tablet e94a73b50b564a97bbee0f5015bb8805 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e4673214ad84f75a861c3af82a13d8f) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.001374 4950 tablet_service.cc:1515] Processing DeleteTablet for tablet 54e68eba747e4a2a947e84e53a444d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.002449 4952 tablet_service.cc:1468] Processing CreateTablet for tablet a319f0cb00f243f08c1cb2f3efd94db1 (DEFAULT_TABLE table=default.Table_d9f785633e4f45eaa38fd0ec36f46224 [id=4796f278305d4f66a36d676736e72b23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:48.003269 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c0338c05a974e0f96865398722c5e8f with delete_type TABLET_DATA_DELETED (replaced by tablet 34b2919e5f534f4295336a4384be1f64) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.003986 6064 raft_consensus.cc:491] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:48.001178 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 54271a1d675e41018c2f2b2a275d761a (DEFAULT_TABLE table=default.Table_90bb38ee5f4c4b3ca1d9f018e53ef6b9 [id=4e809bf42770487ea08e0cc70065a28b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:48.001174 4958 tablet_service.cc:1468] Processing CreateTablet for tablet cdb9c1a7f8d64821b2efc4cdb655382b (DEFAULT_TABLE table=default.Table_65406953a5444c54a77eaed676d03f6c [id=819506362a8e4070b951855ee287fe33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:48.004449 6064 raft_consensus.cc:513] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:48.006217 5366 tablet_bootstrap.cc:654] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:48.006387 6064 leader_election.cc:290] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:48.010124 5936 ts_tablet_manager.cc:1905] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:48.021010 4384 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e9f09f0381eb420ca57c18a5be6f2cc2 consensus state has no leader
I20250626 19:57:48.026877 4397 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet e7c352224d0a46c6a922bf0a0d765f17
W20250626 19:57:48.033071 4399 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 69) took 3503 ms (client timeout 1000 ms). Trace:
I20250626 19:57:48.040191 4397 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34796 (request call id 173) took 2083 ms. Trace:
W20250626 19:57:48.033428 4399 rpcz_store.cc:269] 0626 19:57:44.529838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.532105 (+ 2267us) service_pool.cc:225] Handling call
0626 19:57:44.581455 (+ 49350us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe7915557ee 00007fe78f26c624 00007fe78f26c5b2 00007fe78f26c4e1 00007fe78f26c45c 00007fe78f26b712 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:48.032991 (+3451536us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:45.493796 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:45.494274 (+ 478us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:45.494300 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:45.494316 (+ 16us) tablet.cc:660] Decoding operations
0626 19:57:45.494658 (+ 342us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:45.494720 (+ 62us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:45.494782 (+ 62us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:45.495016 (+ 234us) tablet.cc:699] Row locks acquired
0626 19:57:45.495041 (+ 25us) write_op.cc:260] PREPARE: finished
0626 19:57:45.495205 (+ 164us) write_op.cc:270] Start()
0626 19:57:45.495334 (+ 129us) write_op.cc:276] Timestamp: P: 1750967865495357 usec, L: 0
0626 19:57:45.495364 (+ 30us) op_driver.cc:348] REPLICATION: starting
0626 19:57:45.500138 (+ 4774us) log.cc:844] Serialized 530 byte log entry
0626 19:57:45.514794 (+ 14656us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72b481 00007fe78f7a2a18 00007fe78fc0f811 00007fe78fc0efe8 00007fe78fc13c81 00007fe78fc13c39 00007fe78fc13bc9 00007fe78fc13b91 00007fe78fc12dbd 000000000059bcd4 000000000059bb09
0626 19:57:45.548155 (+ 33361us) op_driver.cc:464] REPLICATION: finished
0626 19:57:45.550029 (+ 1874us) write_op.cc:301] APPLY: starting
0626 19:57:45.550107 (+ 78us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:45.550230 (+ 123us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:45.550261 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:45.550492 (+ 231us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:45.550666 (+ 174us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:45.550734 (+ 68us) write_op.cc:312] APPLY: finished
0626 19:57:45.551011 (+ 277us) log.cc:844] Serialized 23 byte log entry
0626 19:57:45.551204 (+ 193us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:45.551287 (+ 83us) write_op.cc:454] Released schema lock
0626 19:57:45.551331 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:45.551631 (+ 300us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:45.551661 (+ 30us) write_op.cc:454] Released schema lock
Related trace 'op':
0626 19:57:45.952982 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:45.953448 (+ 466us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:45.953486 (+ 38us) write_op.cc:435] Acquired schema lock
0626 19:57:45.953506 (+ 20us) tablet.cc:660] Decoding operations
0626 19:57:45.953888 (+ 382us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:45.953955 (+ 67us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:45.953996 (+ 41us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:45.957252 (+ 3256us) tablet.cc:699] Row locks acquired
0626 19:57:45.957270 (+ 18us) write_op.cc:260] PREPARE: finished
0626 19:57:45.957426 (+ 156us) write_op.cc:270] Start()
0626 19:57:45.957545 (+ 119us) write_op.cc:276] Timestamp: P: 1750967865957562 usec, L: 0
0626 19:57:45.957566 (+ 21us) op_driver.cc:348] REPLICATION: starting
0626 19:57:45.958193 (+ 627us) log.cc:844] Serialized 356 byte log entry
0626 19:57:45.991915 (+ 33722us) op_driver.cc:464] REPLICATION: finished
0626 19:57:45.999247 (+ 7332us) write_op.cc:301] APPLY: starting
0626 19:57:45.999313 (+ 66us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:45.999411 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:45.999512 (+ 101us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:45.999705 (+ 193us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:45.999833 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:45.999876 (+ 43us) write_op.cc:312] APPLY: finished
0626 19:57:46.000051 (+ 175us) log.cc:844] Serialized 23 byte log entry
0626 19:57:46.000171 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:46.000231 (+ 60us) write_op.cc:454] Released schema lock
0626 19:57:46.000260 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:46.000500 (+ 240us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:46.000524 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":62063,"reported_tablets":73,"spinlock_wait_cycles":26142976,"child_traces":[["op",{"apply.queue_time_us":1663,"apply.run_cpu_time_us":1566,"apply.run_wall_time_us":1574,"mutex_wait_us":45088,"num_ops":1,"prepare.queue_time_us":50799,"prepare.run_cpu_time_us":3350,"prepare.run_wall_time_us":21212,"raft.queue_time_us":3733,"raft.run_cpu_time_us":723,"raft.run_wall_time_us":925,"replication_time_us":52646,"spinlock_wait_cycles":9336576}],["op",{"apply.queue_time_us":1126,"apply.run_cpu_time_us":1551,"apply.run_wall_time_us":7277,"mutex_wait_us":96644,"num_ops":1,"prepare.queue_time_us":178717,"prepare.run_cpu_time_us":2502,"prepare.run_wall_time_us":21956,"raft.queue_time_us":15692,"raft.run_cpu_time_us":624,"raft.run_wall_time_us":624,"replication_time_us":34265}]]}
I20250626 19:57:48.041507 4800 tablet_service.cc:1515] Processing DeleteTablet for tablet e7c352224d0a46c6a922bf0a0d765f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:48.044131 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cc96a3e9c174cbca0196c5a1b27455c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.047390 4952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a319f0cb00f243f08c1cb2f3efd94db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:48.049376 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb9c1a7f8d64821b2efc4cdb655382b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:48.050364 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54271a1d675e41018c2f2b2a275d761a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:48.051055 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8399c219d0e4485bb7926eefd78f210" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:48.051184 5907 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 19:57:48.051373 5819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 7 candidate_status { last_received { term: 6 index: 226 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:48.052287 5819 raft_consensus.cc:2466] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae949d1fcdf54e87930871ac0ae7da31 in term 5.
W20250626 19:57:48.052609 4756 leader_election.cc:343] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:48.053031 5819 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59376 (request call id 1) took 3410 ms (client timeout 642 ms). Trace:
W20250626 19:57:48.053418 5819 rpcz_store.cc:269] 0626 19:57:44.642269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.642416 (+ 147us) service_pool.cc:225] Handling call
0626 19:57:48.052857 (+3410441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 19:57:48.058527 5830 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 7 candidate_status { last_received { term: 6 index: 226 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
W20250626 19:57:48.059577 5830 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59376 (request call id 0) took 3828 ms (client timeout 642 ms). Trace:
I20250626 19:57:48.043675 5965 rpcz_store.cc:276] 0626 19:57:46.073628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.073783 (+ 155us) service_pool.cc:225] Handling call
0626 19:57:47.302952 (+1229169us) ts_tablet_manager.cc:1145] Deleting tablet 9172cb6576b240f6b25c3b1267618e6c
0626 19:57:47.303091 (+ 139us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:47.739428 (+436337us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:47.931820 (+192392us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:47.993095 (+ 61275us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56699,"tablet-delete.queue_time_us":1228543}
W20250626 19:57:48.060002 5830 rpcz_store.cc:269] 0626 19:57:44.231147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.231359 (+ 212us) service_pool.cc:225] Handling call
0626 19:57:48.059515 (+3828156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 19:57:48.061977 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c5aba053ca4535ad2376b2066cfe47" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:48.063055 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet f6ec952321924098a142f1438a283aab with delete_type TABLET_DATA_DELETED (replaced by tablet 2d0a30ee16d043f9837dce612dae670a) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.040593 4397 rpcz_store.cc:276] 0626 19:57:45.957076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.016374 (+ 59298us) service_pool.cc:225] Handling call
0626 19:57:46.018020 (+ 1646us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:46.018617 (+ 597us) catalog_manager.cc:2790] Modifying in-memory table state
0626 19:57:46.018962 (+ 345us) catalog_manager.cc:2799] Locking tablets
0626 19:57:46.019526 (+ 564us) catalog_manager.cc:2813] Removing table and tablets from system table
0626 19:57:46.148881 (+129355us) catalog_manager.cc:2831] Removing table from by-name map
0626 19:57:46.292177 (+143296us) catalog_manager.cc:2847] Committing in-memory state
0626 19:57:48.026734 (+1734557us) catalog_manager.cc:2851] Aborting table tasks
0626 19:57:48.040136 (+ 13402us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:46.130991 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:46.131455 (+ 464us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:46.131481 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:46.131493 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:46.131838 (+ 345us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:46.131892 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:46.131922 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:46.132228 (+ 306us) tablet.cc:699] Row locks acquired
0626 19:57:46.132243 (+ 15us) write_op.cc:260] PREPARE: finished
0626 19:57:46.132376 (+ 133us) write_op.cc:270] Start()
0626 19:57:46.132470 (+ 94us) write_op.cc:276] Timestamp: P: 1750967866132392 usec, L: 0
0626 19:57:46.132492 (+ 22us) op_driver.cc:348] REPLICATION: starting
0626 19:57:46.133042 (+ 550us) log.cc:844] Serialized 762 byte log entry
0626 19:57:46.147007 (+ 13965us) op_driver.cc:464] REPLICATION: finished
0626 19:57:46.147368 (+ 361us) write_op.cc:301] APPLY: starting
0626 19:57:46.147428 (+ 60us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:46.147539 (+ 111us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:46.147626 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:46.147931 (+ 305us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:46.148068 (+ 137us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:46.148113 (+ 45us) write_op.cc:312] APPLY: finished
0626 19:57:46.148291 (+ 178us) log.cc:844] Serialized 29 byte log entry
0626 19:57:46.148450 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:46.148515 (+ 65us) write_op.cc:454] Released schema lock
0626 19:57:46.148551 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:46.158375 (+ 9824us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:46.158520 (+ 145us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":222,"apply.run_cpu_time_us":1365,"apply.run_wall_time_us":1365,"num_ops":2,"prepare.queue_time_us":109769,"prepare.run_cpu_time_us":2477,"prepare.run_wall_time_us":27237,"raft.queue_time_us":105,"raft.run_cpu_time_us":878,"raft.run_wall_time_us":877,"replication_time_us":14411}]]}
I20250626 19:57:48.076159 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet 609d36d46bb84017b2a3a7ad48bfccfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.076511 5353 tablet_bootstrap.cc:654] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:48.080190 4756 leader_election.cc:343] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:48.080613 4756 leader_election.cc:304] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:48.081631 6069 raft_consensus.cc:2747] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:48.085430 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:48.093305 5836 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 5 FOLLOWER]: Advancing to term 6
I20250626 19:57:48.094965 4942 tablet_service.cc:1515] Processing DeleteTablet for tablet 1feb6785d0c7459fbc6898b0c9d3e38c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.096606 4943 tablet_service.cc:1515] Processing DeleteTablet for tablet e7c352224d0a46c6a922bf0a0d765f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:41356
W20250626 19:57:48.114858 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 135964us
I20250626 19:57:48.125177 5936 ts_tablet_manager.cc:1918] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:48.125550 5936 log.cc:1199] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/f807cf54bbb146ff989762c18410e6bb
I20250626 19:57:48.126188 5936 ts_tablet_manager.cc:1939] T f807cf54bbb146ff989762c18410e6bb P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:48.136184 4954 tablet_service.cc:1515] Processing DeleteTablet for tablet 65af5ae255d74143976392bccc50f3ba with delete_type TABLET_DATA_DELETED (replaced by tablet a8ef6f024d0642d4a6dd67d646787868) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.138578 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 9172cb6576b240f6b25c3b1267618e6c (table default.Table_2e7897256c7140aeb545751922965c14 [id=441b14db312f4da1be3f2219cf745bc6]) successfully deleted
I20250626 19:57:48.142241 4953 tablet_service.cc:1515] Processing DeleteTablet for tablet fc526a05990446a3b3bc973e79511476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:41356
W20250626 19:57:48.143114 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 260601us
I20250626 19:57:48.143491 4951 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.144057 4956 tablet_service.cc:1515] Processing DeleteTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.151993 4399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_9f7e67f1042843f89daa2817b09e9fb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:48.152675 5965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 200) took 1178 ms. Trace:
I20250626 19:57:48.153118 5965 rpcz_store.cc:276] 0626 19:57:46.974051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.974168 (+ 117us) service_pool.cc:225] Handling call
0626 19:57:48.152234 (+1178066us) ts_tablet_manager.cc:1145] Deleting tablet 56f319c748374b7cbbf037ef55aecd1b
0626 19:57:48.152294 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:48.152607 (+ 313us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1177471}
W20250626 19:57:48.161072 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 56f319c748374b7cbbf037ef55aecd1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56f319c748374b7cbbf037ef55aecd1b already in progress: creating tablet
W20250626 19:57:48.176105 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet e94a73b50b564a97bbee0f5015bb8805 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e94a73b50b564a97bbee0f5015bb8805 already in progress: creating tablet
I20250626 19:57:48.177116 5965 tablet_replica.cc:331] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
W20250626 19:57:48.176721 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 54e68eba747e4a2a947e84e53a444d08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54e68eba747e4a2a947e84e53a444d08 already in progress: creating tablet
I20250626 19:57:48.177850 5965 raft_consensus.cc:2241] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250626 19:57:48.177997 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 4c0338c05a974e0f96865398722c5e8f because the tablet was not found. No further retry: Not found: Tablet not found: 4c0338c05a974e0f96865398722c5e8f
I20250626 19:57:48.178267 5936 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 168) took 2206 ms. Trace:
W20250626 19:57:48.178791 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 3cc96a3e9c174cbca0196c5a1b27455c because the tablet was not found. No further retry: Not found: Tablet not found: 3cc96a3e9c174cbca0196c5a1b27455c
I20250626 19:57:48.178583 5936 rpcz_store.cc:276] 0626 19:57:45.971737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.971874 (+ 137us) service_pool.cc:225] Handling call
0626 19:57:47.894050 (+1922176us) ts_tablet_manager.cc:1145] Deleting tablet f807cf54bbb146ff989762c18410e6bb
0626 19:57:47.894085 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:48.073697 (+179612us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:48.124934 (+ 51237us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:48.178205 (+ 53271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1921581}
W20250626 19:57:48.179479 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet f6ec952321924098a142f1438a283aab because the tablet was not found. No further retry: Not found: Tablet not found: f6ec952321924098a142f1438a283aab
I20250626 19:57:48.179754 5936 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 169) took 2206 ms. Trace:
I20250626 19:57:48.180047 5936 rpcz_store.cc:276] 0626 19:57:45.973377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.973501 (+ 124us) service_pool.cc:225] Handling call
0626 19:57:48.179516 (+2206015us) ts_tablet_manager.cc:1145] Deleting tablet cb34ffcd4da04e32a1220b3f298ec22d
0626 19:57:48.179554 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:48.179704 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2205440}
I20250626 19:57:48.181001 5936 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 170) took 2206 ms. Trace:
W20250626 19:57:48.182192 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 609d36d46bb84017b2a3a7ad48bfccfc because the tablet was not found. No further retry: Not found: Tablet not found: 609d36d46bb84017b2a3a7ad48bfccfc
W20250626 19:57:48.182842 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 1feb6785d0c7459fbc6898b0c9d3e38c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1feb6785d0c7459fbc6898b0c9d3e38c already in progress: creating tablet
I20250626 19:57:48.183481 5836 pending_rounds.cc:85] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Aborting all ops after (but not including) 85
I20250626 19:57:48.183822 5836 pending_rounds.cc:107] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Aborting uncommitted WRITE_OP operation due to leader change: 5.86
I20250626 19:57:48.185089 5836 pending_rounds.cc:107] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Aborting uncommitted WRITE_OP operation due to leader change: 5.87
I20250626 19:57:48.191514 5353 tablet_bootstrap.cc:492] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:48.191939 5353 ts_tablet_manager.cc:1397] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.351s user 0.010s sys 0.009s
W20250626 19:57:48.175155 4399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f7e67f1042843f89daa2817b09e9fb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:48.178303 5965 raft_consensus.cc:2270] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:48.204648 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet f807cf54bbb146ff989762c18410e6bb (table default.Table_f85ad493ebbf4e3b8c7eea1edfcf6a67 [id=fa2e7940491b4fbbac3ee7d93df1476d]) successfully deleted
W20250626 19:57:48.205489 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet cb34ffcd4da04e32a1220b3f298ec22d because the tablet was not found. No further retry: Not found: Tablet not found: cb34ffcd4da04e32a1220b3f298ec22d
I20250626 19:57:48.207854 5965 ts_tablet_manager.cc:1905] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:48.211025 4396 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34796 (request call id 175) took 2243 ms. Trace:
I20250626 19:57:48.181268 5936 rpcz_store.cc:276] 0626 19:57:45.974619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.974742 (+ 123us) service_pool.cc:225] Handling call
0626 19:57:48.180789 (+2206047us) ts_tablet_manager.cc:1145] Deleting tablet 609d36d46bb84017b2a3a7ad48bfccfc
0626 19:57:48.180827 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:48.180956 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2205406}
I20250626 19:57:48.212332 5936 tablet_replica.cc:331] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
W20250626 19:57:48.212354 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 609d36d46bb84017b2a3a7ad48bfccfc because the tablet was not found. No further retry: Not found: Tablet not found: 609d36d46bb84017b2a3a7ad48bfccfc
I20250626 19:57:48.213091 5936 raft_consensus.cc:2241] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250626 19:57:48.209344 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 76458.2us
I20250626 19:57:48.210788 5353 raft_consensus.cc:357] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.221153 5353 raft_consensus.cc:383] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:48.222128 5353 raft_consensus.cc:738] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:48.211376 4396 rpcz_store.cc:276] 0626 19:57:45.967357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.106710 (+139353us) service_pool.cc:225] Handling call
0626 19:57:46.125450 (+ 18740us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe7915573b6 00007fe78f27cca4 00007fe78f27cc32 00007fe78f27cb61 00007fe78f27cadc 00007fe78f27bd92 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:46.134254 (+ 8804us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:46.134872 (+ 618us) catalog_manager.cc:3683] Apply alter schema
0626 19:57:46.269932 (+135060us) catalog_manager.cc:3715] Acquired catalog manager lock
0626 19:57:46.323418 (+ 53486us) catalog_manager.cc:3873] Updating metadata on disk
0626 19:57:46.620801 (+297383us) catalog_manager.cc:3904] Committing alterations to in-memory state
0626 19:57:48.210965 (+1590164us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:46.377595 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:46.378060 (+ 465us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:46.378084 (+ 24us) write_op.cc:435] Acquired schema lock
0626 19:57:46.378097 (+ 13us) tablet.cc:660] Decoding operations
0626 19:57:46.378383 (+ 286us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:46.378429 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:46.378453 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:46.378607 (+ 154us) tablet.cc:699] Row locks acquired
0626 19:57:46.378619 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:46.378732 (+ 113us) write_op.cc:270] Start()
0626 19:57:46.378844 (+ 112us) write_op.cc:276] Timestamp: P: 1750967866378846 usec, L: 0
0626 19:57:46.378859 (+ 15us) op_driver.cc:348] REPLICATION: starting
0626 19:57:46.379382 (+ 523us) log.cc:844] Serialized 405 byte log entry
0626 19:57:46.617212 (+237830us) op_driver.cc:464] REPLICATION: finished
0626 19:57:46.619449 (+ 2237us) write_op.cc:301] APPLY: starting
0626 19:57:46.619495 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:46.619573 (+ 78us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:46.619659 (+ 86us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:46.619787 (+ 128us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:46.619902 (+ 115us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:46.619941 (+ 39us) write_op.cc:312] APPLY: finished
0626 19:57:46.620081 (+ 140us) log.cc:844] Serialized 23 byte log entry
0626 19:57:46.620217 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:46.620271 (+ 54us) write_op.cc:454] Released schema lock
0626 19:57:46.620298 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:46.620496 (+ 198us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:46.620523 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":40972160,"child_traces":[["op",{"apply.queue_time_us":2116,"apply.run_cpu_time_us":1020,"apply.run_wall_time_us":1020,"mutex_wait_us":51125,"num_ops":1,"prepare.queue_time_us":51236,"prepare.run_cpu_time_us":1995,"prepare.run_wall_time_us":1995,"replication_time_us":238284}]]}
I20250626 19:57:48.228878 4396 master_service.cc:948] ReplaceTablet: received request to replace tablet 3143181fae16411888d74f9152f3d5cc from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:48.239811 5353 consensus_queue.cc:260] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:48.254834 6088 log.cc:927] Time spent T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20250626 19:57:48.256693 5366 tablet_bootstrap.cc:492] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:48.257228 5366 ts_tablet_manager.cc:1397] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.273s user 0.004s sys 0.006s
I20250626 19:57:48.259586 5366 raft_consensus.cc:357] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.262610 5366 raft_consensus.cc:383] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:48.265241 5366 raft_consensus.cc:738] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:48.271060 4800 tablet_service.cc:1515] Processing DeleteTablet for tablet e94a73b50b564a97bbee0f5015bb8805 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e4673214ad84f75a861c3af82a13d8f) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:48.271632 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cc96a3e9c174cbca0196c5a1b27455c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:48.271818 5353 ts_tablet_manager.cc:1428] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.080s user 0.006s sys 0.001s
I20250626 19:57:48.268378 5366 consensus_queue.cc:260] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.282255 5353 tablet_bootstrap.cc:492] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:48.287683 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:48.317215 5937 ts_tablet_manager.cc:1918] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:48.322299 5937 log.cc:1199] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/609d36d46bb84017b2a3a7ad48bfccfc
I20250626 19:57:48.323535 5937 ts_tablet_manager.cc:1939] T 609d36d46bb84017b2a3a7ad48bfccfc P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:48.325927 6074 raft_consensus.cc:491] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:48.326489 6074 raft_consensus.cc:513] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.337930 4394 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34796 (request call id 180) took 2004 ms. Trace:
I20250626 19:57:48.338255 6074 leader_election.cc:290] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:48.338333 4394 rpcz_store.cc:276] 0626 19:57:46.333974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.373776 (+ 39802us) service_pool.cc:225] Handling call
0626 19:57:46.374772 (+ 996us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:46.376678 (+ 1906us) catalog_manager.cc:3683] Apply alter schema
0626 19:57:46.915621 (+538943us) catalog_manager.cc:3715] Acquired catalog manager lock
0626 19:57:46.915875 (+ 254us) catalog_manager.cc:3873] Updating metadata on disk
0626 19:57:46.935941 (+ 20066us) catalog_manager.cc:3904] Committing alterations to in-memory state
0626 19:57:48.337867 (+1401926us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:46.917294 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:46.917714 (+ 420us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:46.917737 (+ 23us) write_op.cc:435] Acquired schema lock
0626 19:57:46.917749 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:46.917996 (+ 247us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:46.918033 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:46.918052 (+ 19us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:46.918183 (+ 131us) tablet.cc:699] Row locks acquired
0626 19:57:46.918194 (+ 11us) write_op.cc:260] PREPARE: finished
0626 19:57:46.918302 (+ 108us) write_op.cc:270] Start()
0626 19:57:46.918378 (+ 76us) write_op.cc:276] Timestamp: P: 1750967866918421 usec, L: 0
0626 19:57:46.918393 (+ 15us) op_driver.cc:348] REPLICATION: starting
0626 19:57:46.918870 (+ 477us) log.cc:844] Serialized 405 byte log entry
0626 19:57:46.933301 (+ 14431us) op_driver.cc:464] REPLICATION: finished
0626 19:57:46.933643 (+ 342us) write_op.cc:301] APPLY: starting
0626 19:57:46.933696 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:46.933786 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:46.933864 (+ 78us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:46.934020 (+ 156us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:46.934151 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:46.934196 (+ 45us) write_op.cc:312] APPLY: finished
0626 19:57:46.934360 (+ 164us) log.cc:844] Serialized 23 byte log entry
0626 19:57:46.934501 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:46.934573 (+ 72us) write_op.cc:454] Released schema lock
0626 19:57:46.934608 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:46.935676 (+ 1068us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:46.935701 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5,"child_traces":[["op",{"apply.queue_time_us":187,"apply.run_cpu_time_us":1221,"apply.run_wall_time_us":1996,"num_ops":1,"prepare.queue_time_us":326,"prepare.run_cpu_time_us":1834,"prepare.run_wall_time_us":1834,"raft.queue_time_us":4892,"raft.run_cpu_time_us":609,"raft.run_wall_time_us":609,"replication_time_us":14802}]]}
I20250626 19:57:48.355124 5937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 184) took 2116 ms. Trace:
I20250626 19:57:48.355592 5937 rpcz_store.cc:276] 0626 19:57:46.238253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.239231 (+ 978us) service_pool.cc:225] Handling call
0626 19:57:47.803500 (+1564269us) ts_tablet_manager.cc:1145] Deleting tablet 609d36d46bb84017b2a3a7ad48bfccfc
0626 19:57:47.803540 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:48.224216 (+420676us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:48.306531 (+ 82315us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:48.355005 (+ 48474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1561818}
I20250626 19:57:48.357812 5937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49550 (request call id 196) took 1383 ms. Trace:
I20250626 19:57:48.363596 4394 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_6635c438f840403e8a1ef43d005d2dad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:48.378840 5353 tablet_bootstrap.cc:654] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:48.358201 5937 rpcz_store.cc:276] 0626 19:57:46.974249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.975674 (+ 1425us) service_pool.cc:225] Handling call
0626 19:57:48.357242 (+1381568us) ts_tablet_manager.cc:1145] Deleting tablet 56f319c748374b7cbbf037ef55aecd1b
0626 19:57:48.357345 (+ 103us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:48.357726 (+ 381us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1380985}
I20250626 19:57:48.390064 6080 consensus_queue.cc:1035] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:57:48.399682 4394 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6635c438f840403e8a1ef43d005d2dad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:48.403695 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83f0431638144710b598b3f87d9a576e" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:48.404166 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8399c219d0e4485bb7926eefd78f210" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:48.404342 4675 raft_consensus.cc:2466] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:48.416396 6080 raft_consensus.cc:491] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:48.416873 6080 raft_consensus.cc:513] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.417455 5410 tablet_bootstrap.cc:492] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:48.417901 5410 ts_tablet_manager.cc:1397] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.715s user 0.011s sys 0.008s
I20250626 19:57:48.425385 5965 ts_tablet_manager.cc:1918] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:48.425765 5965 log.cc:1199] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/e7c352224d0a46c6a922bf0a0d765f17
I20250626 19:57:48.426541 5965 ts_tablet_manager.cc:1939] T e7c352224d0a46c6a922bf0a0d765f17 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:48.451221 5353 tablet_bootstrap.cc:492] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:48.451668 5353 ts_tablet_manager.cc:1397] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.170s user 0.010s sys 0.004s
I20250626 19:57:48.455164 5353 raft_consensus.cc:357] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.455749 5353 raft_consensus.cc:383] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:48.456004 5353 raft_consensus.cc:738] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:48.454650 5410 raft_consensus.cc:357] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.456811 5410 raft_consensus.cc:383] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:48.461977 5410 raft_consensus.cc:738] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:48.463390 5410 consensus_queue.cc:260] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.465452 5410 ts_tablet_manager.cc:1428] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.047s user 0.006s sys 0.001s
I20250626 19:57:48.470156 5353 consensus_queue.cc:260] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.470508 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet e7c352224d0a46c6a922bf0a0d765f17 (table default.Table_822bad36b81c42ac9101bcab4aaca9a4 [id=90440da2b42248e98586aa690f9199bd]) successfully deleted
W20250626 19:57:48.471776 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 65af5ae255d74143976392bccc50f3ba because the tablet was not found. No further retry: Not found: Tablet not found: 65af5ae255d74143976392bccc50f3ba
W20250626 19:57:48.472344 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet fc526a05990446a3b3bc973e79511476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc526a05990446a3b3bc973e79511476 already in progress: creating tablet
W20250626 19:57:48.473480 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet f807cf54bbb146ff989762c18410e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f807cf54bbb146ff989762c18410e6bb already in progress: creating tablet
I20250626 19:57:48.473636 5410 tablet_bootstrap.cc:492] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
W20250626 19:57:48.474043 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet cb34ffcd4da04e32a1220b3f298ec22d because the tablet was not found. No further retry: Not found: Tablet not found: cb34ffcd4da04e32a1220b3f298ec22d
I20250626 19:57:48.483322 5353 ts_tablet_manager.cc:1428] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.031s user 0.004s sys 0.002s
I20250626 19:57:48.484171 5353 tablet_bootstrap.cc:492] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:48.487620 5936 raft_consensus.cc:2270] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:48.489236 5936 ts_tablet_manager.cc:1905] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:48.497491 6117 tablet.cc:1722] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:48.501152 5366 ts_tablet_manager.cc:1428] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.244s user 0.003s sys 0.003s
I20250626 19:57:48.502347 5366 tablet_bootstrap.cc:492] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
W20250626 19:57:48.507488 4756 consensus_peers.cc:487] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:48.509179 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 243195us
I20250626 19:57:48.521966 5936 ts_tablet_manager.cc:1918] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:48.522341 5936 log.cc:1199] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/9172cb6576b240f6b25c3b1267618e6c
I20250626 19:57:48.526715 5936 ts_tablet_manager.cc:1939] T 9172cb6576b240f6b25c3b1267618e6c P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:48.527731 5936 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 171) took 2450 ms. Trace:
I20250626 19:57:48.528040 5936 rpcz_store.cc:276] 0626 19:57:46.077022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.077843 (+ 821us) service_pool.cc:225] Handling call
0626 19:57:48.212185 (+2134342us) ts_tablet_manager.cc:1145] Deleting tablet 9172cb6576b240f6b25c3b1267618e6c
0626 19:57:48.212230 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:48.499257 (+287027us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:48.521748 (+ 22491us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:48.527675 (+ 5927us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":274039,"tablet-delete.queue_time_us":2133810}
I20250626 19:57:48.528930 6069 raft_consensus.cc:491] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:48.529276 5936 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40742 (request call id 182) took 1544 ms. Trace:
I20250626 19:57:48.529372 6069 raft_consensus.cc:513] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.529606 5936 rpcz_store.cc:276] 0626 19:57:46.984421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.984533 (+ 112us) service_pool.cc:225] Handling call
0626 19:57:48.528956 (+1544423us) ts_tablet_manager.cc:1145] Deleting tablet 56f319c748374b7cbbf037ef55aecd1b
0626 19:57:48.528994 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:48.529229 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1543902}
I20250626 19:57:48.531060 5936 tablet_replica.cc:331] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:48.531512 6069 leader_election.cc:290] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:48.531805 5936 raft_consensus.cc:2241] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:48.532485 5936 raft_consensus.cc:2270] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:48.534518 5936 ts_tablet_manager.cc:1905] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:48.541023 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 609d36d46bb84017b2a3a7ad48bfccfc (table default.Table_d23ee99af7b845b78d2200b962f681a6 [id=fc725d6df6774d4e922e2d98d461b111]) successfully deleted
W20250626 19:57:48.557850 4390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 84) took 3888 ms (client timeout 1000 ms). Trace:
W20250626 19:57:48.566995 4395 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 79) took 2785 ms (client timeout 1000 ms). Trace:
I20250626 19:57:48.572088 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 9172cb6576b240f6b25c3b1267618e6c (table default.Table_2e7897256c7140aeb545751922965c14 [id=441b14db312f4da1be3f2219cf745bc6]) successfully deleted
W20250626 19:57:48.572782 4383 catalog_manager.cc:4908] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 56f319c748374b7cbbf037ef55aecd1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56f319c748374b7cbbf037ef55aecd1b already in progress: creating tablet
W20250626 19:57:48.573307 4383 catalog_manager.cc:4908] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 54e68eba747e4a2a947e84e53a444d08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54e68eba747e4a2a947e84e53a444d08 already in progress: creating tablet
I20250626 19:57:48.514308 6048 raft_consensus.cc:491] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:48.525444 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83f0431638144710b598b3f87d9a576e" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:48.539835 5353 tablet_bootstrap.cc:654] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:48.519162 6080 leader_election.cc:290] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:48.558835 6125 tablet.cc:1722] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250626 19:57:48.558184 4390 rpcz_store.cc:269] 0626 19:57:44.669840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:44.669996 (+ 156us) service_pool.cc:225] Handling call
0626 19:57:48.557778 (+3887782us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:45.862391 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:45.862908 (+ 517us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:45.862938 (+ 30us) write_op.cc:435] Acquired schema lock
0626 19:57:45.862951 (+ 13us) tablet.cc:660] Decoding operations
0626 19:57:45.863244 (+ 293us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:45.863294 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:45.863324 (+ 30us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:45.863484 (+ 160us) tablet.cc:699] Row locks acquired
0626 19:57:45.863496 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:45.863621 (+ 125us) write_op.cc:270] Start()
0626 19:57:45.863714 (+ 93us) write_op.cc:276] Timestamp: P: 1750967865863723 usec, L: 0
0626 19:57:45.863731 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:45.864319 (+ 588us) log.cc:844] Serialized 530 byte log entry
0626 19:57:45.883032 (+ 18713us) op_driver.cc:464] REPLICATION: finished
0626 19:57:45.883280 (+ 248us) write_op.cc:301] APPLY: starting
0626 19:57:45.883331 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:45.883424 (+ 93us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:45.883523 (+ 99us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:45.883690 (+ 167us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:45.883850 (+ 160us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:45.883890 (+ 40us) write_op.cc:312] APPLY: finished
0626 19:57:45.884055 (+ 165us) log.cc:844] Serialized 23 byte log entry
0626 19:57:45.884200 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:45.884259 (+ 59us) write_op.cc:454] Released schema lock
0626 19:57:45.884289 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:45.884515 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:45.884537 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":10903,"reported_tablets":74,"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":1225,"apply.run_wall_time_us":1224,"num_ops":1,"prepare.queue_time_us":121165,"prepare.run_cpu_time_us":2240,"prepare.run_wall_time_us":2239,"raft.queue_time_us":908,"raft.run_cpu_time_us":643,"raft.run_wall_time_us":643,"replication_time_us":19175}]]}
I20250626 19:57:48.581282 6048 raft_consensus.cc:513] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.594209 6048 leader_election.cc:290] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:48.597193 6048 raft_consensus.cc:491] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:48.597608 6048 raft_consensus.cc:513] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.600803 6048 leader_election.cc:290] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:48.608309 4395 rpcz_store.cc:269] 0626 19:57:45.781582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.827441 (+ 45859us) service_pool.cc:225] Handling call
0626 19:57:48.566932 (+2739491us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":100940,"reported_tablets":70}
I20250626 19:57:48.550093 4801 tablet_service.cc:1515] Processing DeleteTablet for tablet f6ec952321924098a142f1438a283aab with delete_type TABLET_DATA_DELETED (replaced by tablet 2d0a30ee16d043f9837dce612dae670a) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:48.551190 4800 tablet_service.cc:1515] Processing DeleteTablet for tablet fc526a05990446a3b3bc973e79511476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:48.552163 4793 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:48.563680 4755 leader_election.cc:304] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
W20250626 19:57:48.570204 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 60498us
W20250626 19:57:48.615013 4755 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 51432.2us
W20250626 19:57:48.615684 4755 leader_election.cc:343] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:48.616113 4755 leader_election.cc:304] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:48.615077 6122 raft_consensus.cc:2802] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:48.573426 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 65af5ae255d74143976392bccc50f3ba with delete_type TABLET_DATA_DELETED (replaced by tablet a8ef6f024d0642d4a6dd67d646787868) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:48.618037 6122 raft_consensus.cc:491] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:48.618417 6119 raft_consensus.cc:2747] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:48.574043 4796 tablet_service.cc:1515] Processing DeleteTablet for tablet 1feb6785d0c7459fbc6898b0c9d3e38c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:48.618556 6122 raft_consensus.cc:3058] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:48.575338 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83f0431638144710b598b3f87d9a576e" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
W20250626 19:57:48.622072 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 56f319c748374b7cbbf037ef55aecd1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56f319c748374b7cbbf037ef55aecd1b already in progress: creating tablet
I20250626 19:57:48.569260 6064 raft_consensus.cc:491] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:48.583702 5366 tablet_bootstrap.cc:654] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:48.624285 6064 raft_consensus.cc:513] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.626076 6064 leader_election.cc:290] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:48.626911 4814 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8be634cb36d40dca3f9338dab93ad2c" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:48.629005 4978 raft_consensus.cc:3058] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:48.628871 4812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4aee96ad3454d65b974c867605c4cdc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:48.629406 4812 raft_consensus.cc:2466] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:48.630733 4657 tablet_service.cc:1468] Processing CreateTablet for tablet cdb9c1a7f8d64821b2efc4cdb655382b (DEFAULT_TABLE table=default.Table_65406953a5444c54a77eaed676d03f6c [id=819506362a8e4070b951855ee287fe33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:48.631989 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb9c1a7f8d64821b2efc4cdb655382b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:48.634452 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4aee96ad3454d65b974c867605c4cdc" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:48.639254 4976 raft_consensus.cc:2466] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:48.645108 6122 raft_consensus.cc:513] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.647969 4649 tablet_service.cc:1468] Processing CreateTablet for tablet 54271a1d675e41018c2f2b2a275d761a (DEFAULT_TABLE table=default.Table_90bb38ee5f4c4b3ca1d9f018e53ef6b9 [id=4e809bf42770487ea08e0cc70065a28b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:48.649245 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54271a1d675e41018c2f2b2a275d761a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:48.651342 6077 consensus_queue.cc:1035] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:48.656968 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4aee96ad3454d65b974c867605c4cdc" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:48.660800 5410 tablet_bootstrap.cc:654] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:48.668696 4911 leader_election.cc:343] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Not found: Tablet not found: e94a73b50b564a97bbee0f5015bb8805
I20250626 19:57:48.669287 4911 leader_election.cc:304] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:48.670393 6077 raft_consensus.cc:2802] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:48.670780 6077 raft_consensus.cc:491] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:48.671152 6077 raft_consensus.cc:3058] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:48.671309 6125 tablet.cc:1722] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:48.682690 4816 raft_consensus.cc:2391] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 93f885c6802341c889a40e9c20fbfff1 in this term.
I20250626 19:57:48.687336 6122 leader_election.cc:290] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
W20250626 19:57:48.666079 4755 leader_election.cc:343] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:48.680543 5366 tablet_bootstrap.cc:492] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:48.692533 5366 ts_tablet_manager.cc:1397] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.190s user 0.010s sys 0.007s
I20250626 19:57:48.694841 5366 raft_consensus.cc:357] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.695813 5366 raft_consensus.cc:383] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:48.696101 5366 raft_consensus.cc:738] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:48.696990 5366 consensus_queue.cc:260] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.646116 6121 consensus_queue.cc:1035] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:48.700627 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8be634cb36d40dca3f9338dab93ad2c" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:48.703155 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b852dd204f8a42cca629901723fc9247" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:48.703926 5366 ts_tablet_manager.cc:1428] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20250626 19:57:48.680974 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b852dd204f8a42cca629901723fc9247" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:48.704641 6137 tablet.cc:1722] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250626 19:57:48.708302 4391 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 34b2919e5f534f4295336a4384be1f64 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 34b2919e5f534f4295336a4384be1f64 consensus state has no leader
I20250626 19:57:48.708869 4756 leader_election.cc:304] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:48.709852 6122 raft_consensus.cc:2802] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:48.710170 6122 raft_consensus.cc:491] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:48.710493 6122 raft_consensus.cc:3058] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:48.711768 4672 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83f0431638144710b598b3f87d9a576e" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:48.712520 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cc96a3e9c174cbca0196c5a1b27455c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:41356
W20250626 19:57:48.713397 4755 leader_election.cc:343] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:48.714378 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 3cc96a3e9c174cbca0196c5a1b27455c because the tablet was not found. No further retry: Not found: Tablet not found: 3cc96a3e9c174cbca0196c5a1b27455c
I20250626 19:57:48.718051 5366 tablet_bootstrap.cc:492] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:48.723732 5366 tablet_bootstrap.cc:654] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:48.723724 4976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83f0431638144710b598b3f87d9a576e" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
W20250626 19:57:48.727310 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 58094.3us
I20250626 19:57:48.712353 4672 raft_consensus.cc:3058] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:48.741324 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e94a73b50b564a97bbee0f5015bb8805" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:48.741494 6144 tablet.cc:1722] T e0bcbb044d1549c88e5b83eeb5c709af P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:48.746312 4672 raft_consensus.cc:2466] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
W20250626 19:57:48.746968 4756 leader_election.cc:343] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:48.747488 4756 leader_election.cc:304] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:48.748589 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet 54e68eba747e4a2a947e84e53a444d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:48.749446 4655 tablet_service.cc:1468] Processing CreateTablet for tablet a319f0cb00f243f08c1cb2f3efd94db1 (DEFAULT_TABLE table=default.Table_d9f785633e4f45eaa38fd0ec36f46224 [id=4796f278305d4f66a36d676736e72b23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:48.750113 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cc96a3e9c174cbca0196c5a1b27455c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:48.750525 4655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a319f0cb00f243f08c1cb2f3efd94db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:48.751137 4651 tablet_service.cc:1515] Processing DeleteTablet for tablet e7c352224d0a46c6a922bf0a0d765f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:48.751307 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet f6ec952321924098a142f1438a283aab with delete_type TABLET_DATA_DELETED (replaced by tablet 2d0a30ee16d043f9837dce612dae670a) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:48.751928 5937 tablet_replica.cc:331] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:48.752550 4653 tablet_service.cc:1515] Processing DeleteTablet for tablet cb34ffcd4da04e32a1220b3f298ec22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:37 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:48.752581 4654 tablet_service.cc:1515] Processing DeleteTablet for tablet e94a73b50b564a97bbee0f5015bb8805 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e4673214ad84f75a861c3af82a13d8f) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:48.753167 6119 raft_consensus.cc:2747] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:48.753496 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 36997dab0ab3430bba63a90805759b30 with delete_type TABLET_DATA_DELETED (replaced by tablet fafbf98e7f4544b6ad96c1c5ab6c82bc) from {username='slave'} at 127.0.0.1:49550
W20250626 19:57:48.754149 4756 leader_election.cc:343] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:48.754323 4648 tablet_service.cc:1515] Processing DeleteTablet for tablet fc526a05990446a3b3bc973e79511476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:48.751880 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c0338c05a974e0f96865398722c5e8f with delete_type TABLET_DATA_DELETED (replaced by tablet 34b2919e5f534f4295336a4384be1f64) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:48.760134 6119 consensus_queue.cc:1035] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:48.761579 5936 ts_tablet_manager.cc:1918] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:48.761948 5936 log.cc:1199] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/e7c352224d0a46c6a922bf0a0d765f17
I20250626 19:57:48.762614 5936 ts_tablet_manager.cc:1939] T e7c352224d0a46c6a922bf0a0d765f17 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:48.763032 4651 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:48.771106 4650 tablet_service.cc:1515] Processing DeleteTablet for tablet 65af5ae255d74143976392bccc50f3ba with delete_type TABLET_DATA_DELETED (replaced by tablet a8ef6f024d0642d4a6dd67d646787868) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:48.771847 4652 tablet_service.cc:1515] Processing DeleteTablet for tablet 9172cb6576b240f6b25c3b1267618e6c with delete_type TABLET_DATA_DELETED (replaced by tablet 290232b0933b451c9d7744165bd72796) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:48.774942 4648 tablet_service.cc:1515] Processing DeleteTablet for tablet 609d36d46bb84017b2a3a7ad48bfccfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:48.770681 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f319c748374b7cbbf037ef55aecd1b with delete_type TABLET_DATA_DELETED (replaced by tablet 15aade4064d742488ce29145185551a7) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:48.776034 4651 tablet_service.cc:1515] Processing DeleteTablet for tablet 1feb6785d0c7459fbc6898b0c9d3e38c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:48.776319 5937 raft_consensus.cc:2241] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:48.776852 5937 raft_consensus.cc:2270] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:48.784003 4399 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 184) took 2168 ms. Trace:
I20250626 19:57:48.784123 6122 raft_consensus.cc:513] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.785324 6147 tablet.cc:1722] T 0f834922c9de46dca48c47748913b11a P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:48.786664 6122 leader_election.cc:290] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:48.784336 4399 rpcz_store.cc:276] 0626 19:57:46.615983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:48.151458 (+1535475us) service_pool.cc:225] Handling call
0626 19:57:48.382998 (+231540us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:48.386081 (+ 3083us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:48.783387 (+397306us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:48.783683 (+ 296us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:48.783939 (+ 256us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:48.409673 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:48.410122 (+ 449us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:48.410148 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:48.410160 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:48.410508 (+ 348us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:48.410572 (+ 64us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:48.410610 (+ 38us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:48.410926 (+ 316us) tablet.cc:699] Row locks acquired
0626 19:57:48.410945 (+ 19us) write_op.cc:260] PREPARE: finished
0626 19:57:48.411090 (+ 145us) write_op.cc:270] Start()
0626 19:57:48.411205 (+ 115us) write_op.cc:276] Timestamp: P: 1750967868411179 usec, L: 0
0626 19:57:48.411231 (+ 26us) op_driver.cc:348] REPLICATION: starting
0626 19:57:48.411914 (+ 683us) log.cc:844] Serialized 433 byte log entry
0626 19:57:48.562819 (+150905us) op_driver.cc:464] REPLICATION: finished
0626 19:57:48.678081 (+115262us) write_op.cc:301] APPLY: starting
0626 19:57:48.678140 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:48.678242 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:48.678331 (+ 89us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:48.678789 (+ 458us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:48.678948 (+ 159us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:48.678991 (+ 43us) write_op.cc:312] APPLY: finished
0626 19:57:48.679177 (+ 186us) log.cc:844] Serialized 29 byte log entry
0626 19:57:48.766167 (+ 86990us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:48.766233 (+ 66us) write_op.cc:454] Released schema lock
0626 19:57:48.766264 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":115070,"num_ops":2,"prepare.queue_time_us":21654,"prepare.run_cpu_time_us":2616,"prepare.run_wall_time_us":25690,"raft.queue_time_us":378,"raft.run_cpu_time_us":652,"raft.run_wall_time_us":651,"replication_time_us":151451,"thread_start_us":157436,"threads_started":2}]]}
I20250626 19:57:48.788578 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4aee96ad3454d65b974c867605c4cdc" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:48.790498 4396 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 3143181fae16411888d74f9152f3d5cc
I20250626 19:57:48.791939 4396 catalog_manager.cc:6652] ReplaceTablet: tablet 3143181fae16411888d74f9152f3d5cc deleted and replaced by tablet 08045a939eed4ec69b49d68b555bdf80
I20250626 19:57:48.792255 4396 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34796 (request call id 185) took 1857 ms. Trace:
I20250626 19:57:48.792526 4396 rpcz_store.cc:276] 0626 19:57:46.934926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:48.228748 (+1293822us) service_pool.cc:225] Handling call
0626 19:57:48.792202 (+563454us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:48.435481 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:48.435938 (+ 457us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:48.435965 (+ 27us) write_op.cc:435] Acquired schema lock
0626 19:57:48.435986 (+ 21us) tablet.cc:660] Decoding operations
0626 19:57:48.441865 (+ 5879us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:48.441910 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:48.441936 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:48.442165 (+ 229us) tablet.cc:699] Row locks acquired
0626 19:57:48.442178 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:57:48.442335 (+ 157us) write_op.cc:270] Start()
0626 19:57:48.442457 (+ 122us) write_op.cc:276] Timestamp: P: 1750967868442433 usec, L: 0
0626 19:57:48.442485 (+ 28us) op_driver.cc:348] REPLICATION: starting
0626 19:57:48.443190 (+ 705us) log.cc:844] Serialized 659 byte log entry
0626 19:57:48.696394 (+253204us) op_driver.cc:464] REPLICATION: finished
0626 19:57:48.725111 (+ 28717us) write_op.cc:301] APPLY: starting
0626 19:57:48.725174 (+ 63us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:48.725282 (+ 108us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:48.725388 (+ 106us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:48.725735 (+ 347us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:48.725960 (+ 225us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:48.726017 (+ 57us) write_op.cc:312] APPLY: finished
0626 19:57:48.726220 (+ 203us) log.cc:844] Serialized 29 byte log entry
0626 19:57:48.726368 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:48.726436 (+ 68us) write_op.cc:454] Released schema lock
0626 19:57:48.726466 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":28540,"num_ops":2,"prepare.queue_time_us":32260,"prepare.run_cpu_time_us":2480,"prepare.run_wall_time_us":11138,"replication_time_us":253795,"thread_start_us":43403,"threads_started":1}]]}
I20250626 19:57:48.795964 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 3143181fae16411888d74f9152f3d5cc with delete_type TABLET_DATA_DELETED (replaced by tablet 08045a939eed4ec69b49d68b555bdf80) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.796631 5965 tablet_replica.cc:331] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:48.797442 5965 raft_consensus.cc:2241] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:48.798242 5965 raft_consensus.cc:2270] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:48.800122 5965 ts_tablet_manager.cc:1905] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:48.800236 4394 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 186) took 1665 ms. Trace:
W20250626 19:57:48.800421 6136 log.cc:927] Time spent T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20250626 19:57:48.802042 6077 raft_consensus.cc:513] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.800599 4394 rpcz_store.cc:276] 0626 19:57:47.134619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:48.362997 (+1228378us) service_pool.cc:225] Handling call
0626 19:57:48.400502 (+ 37505us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:48.407272 (+ 6770us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:48.799658 (+392386us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:48.799935 (+ 277us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:48.800183 (+ 248us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:48.446825 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:48.449460 (+ 2635us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:48.449488 (+ 28us) write_op.cc:435] Acquired schema lock
0626 19:57:48.449505 (+ 17us) tablet.cc:660] Decoding operations
0626 19:57:48.449837 (+ 332us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:48.449880 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:48.449908 (+ 28us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:48.450120 (+ 212us) tablet.cc:699] Row locks acquired
0626 19:57:48.450137 (+ 17us) write_op.cc:260] PREPARE: finished
0626 19:57:48.450254 (+ 117us) write_op.cc:270] Start()
0626 19:57:48.450343 (+ 89us) write_op.cc:276] Timestamp: P: 1750967868450367 usec, L: 0
0626 19:57:48.450357 (+ 14us) op_driver.cc:348] REPLICATION: starting
0626 19:57:48.450943 (+ 586us) log.cc:844] Serialized 433 byte log entry
0626 19:57:48.740177 (+289234us) op_driver.cc:464] REPLICATION: finished
0626 19:57:48.798042 (+ 57865us) write_op.cc:301] APPLY: starting
0626 19:57:48.798093 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:48.798304 (+ 211us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:48.798385 (+ 81us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:48.798808 (+ 423us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:48.798942 (+ 134us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:48.798984 (+ 42us) write_op.cc:312] APPLY: finished
0626 19:57:48.799177 (+ 193us) log.cc:844] Serialized 29 byte log entry
0626 19:57:48.799373 (+ 196us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:48.799434 (+ 61us) write_op.cc:454] Released schema lock
0626 19:57:48.799479 (+ 45us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":57730,"num_ops":2,"prepare.queue_time_us":38354,"prepare.run_cpu_time_us":2265,"prepare.run_wall_time_us":10887,"replication_time_us":289721,"threads_started":1}]]}
I20250626 19:57:48.803086 4979 raft_consensus.cc:2391] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 93f885c6802341c889a40e9c20fbfff1 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:48.767808 4755 leader_election.cc:304] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:48.803476 4394 master_service.cc:948] ReplaceTablet: received request to replace tablet c990e4eca5cf4a8e8295e9599cf5be29 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:48.803882 6122 raft_consensus.cc:2802] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:48.803915 6077 leader_election.cc:290] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:48.804625 6122 raft_consensus.cc:695] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:48.812978 4396 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_a1bd6779cf0e482aba51a93dd289d342"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:48.821900 4395 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_3129bcf22b744b0d8072d7e291616bc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:48.823081 6122 consensus_queue.cc:237] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.826977 5366 tablet_bootstrap.cc:492] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:48.827368 5366 ts_tablet_manager.cc:1397] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.109s user 0.009s sys 0.000s
I20250626 19:57:48.829993 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4aee96ad3454d65b974c867605c4cdc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:48.830734 4821 raft_consensus.cc:2391] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 93f885c6802341c889a40e9c20fbfff1 in this term.
I20250626 19:57:48.831390 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet e7c352224d0a46c6a922bf0a0d765f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.832544 6139 tablet.cc:1722] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:48.840658 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet e94a73b50b564a97bbee0f5015bb8805 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e4673214ad84f75a861c3af82a13d8f) from {username='slave'} at 127.0.0.1:41356
W20250626 19:57:48.815291 4754 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.4.46.188:40801, user_credentials={real_user=slave}} blocked reactor thread for 78256.6us
W20250626 19:57:48.710525 4391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 85) took 2399 ms (client timeout 1000 ms). Trace:
W20250626 19:57:48.815944 4755 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 48261us
I20250626 19:57:48.842149 4796 tablet_service.cc:1515] Processing DeleteTablet for tablet 3143181fae16411888d74f9152f3d5cc with delete_type TABLET_DATA_DELETED (replaced by tablet 08045a939eed4ec69b49d68b555bdf80) from {username='slave'} at 127.0.0.1:40742
W20250626 19:57:48.845623 4395 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3129bcf22b744b0d8072d7e291616bc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:57:48.846501 4396 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1bd6779cf0e482aba51a93dd289d342 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:48.849308 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet e7c352224d0a46c6a922bf0a0d765f17 (table default.Table_822bad36b81c42ac9101bcab4aaca9a4 [id=90440da2b42248e98586aa690f9199bd]) successfully deleted
I20250626 19:57:48.850118 5936 tablet_replica.cc:331] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:48.850916 5936 raft_consensus.cc:2241] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:48.851377 5936 raft_consensus.cc:2270] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:48.852028 5353 tablet_bootstrap.cc:492] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:48.852471 5353 ts_tablet_manager.cc:1397] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.368s user 0.011s sys 0.013s
I20250626 19:57:48.853106 5936 ts_tablet_manager.cc:1905] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:48.853171 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet e94a73b50b564a97bbee0f5015bb8805 because the tablet was not found. No further retry: Not found: Tablet not found: e94a73b50b564a97bbee0f5015bb8805
W20250626 19:57:48.857000 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 3cc96a3e9c174cbca0196c5a1b27455c because the tablet was not found. No further retry: Not found: Tablet not found: 3cc96a3e9c174cbca0196c5a1b27455c
I20250626 19:57:48.862675 6152 tablet.cc:1722] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250626 19:57:48.841912 4391 rpcz_store.cc:269] 0626 19:57:46.310861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:46.318692 (+ 7831us) service_pool.cc:225] Handling call
0626 19:57:47.682044 (+1363352us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:47.682067 (+ 23us) write_op.cc:454] Released schema lock
0626 19:57:48.710459 (+1028392us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:47.662682 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:47.663099 (+ 417us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:47.663125 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:47.663137 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:47.663475 (+ 338us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:47.663514 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:47.663538 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:47.663787 (+ 249us) tablet.cc:699] Row locks acquired
0626 19:57:47.663799 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:57:47.663906 (+ 107us) write_op.cc:270] Start()
0626 19:57:47.663986 (+ 80us) write_op.cc:276] Timestamp: P: 1750967867663922 usec, L: 0
0626 19:57:47.664002 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:57:47.664509 (+ 507us) log.cc:844] Serialized 907 byte log entry
0626 19:57:47.672987 (+ 8478us) op_driver.cc:464] REPLICATION: finished
0626 19:57:47.673744 (+ 757us) write_op.cc:301] APPLY: starting
0626 19:57:47.673804 (+ 60us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:47.673921 (+ 117us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:47.674010 (+ 89us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:47.674273 (+ 263us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:47.674410 (+ 137us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:47.674457 (+ 47us) write_op.cc:312] APPLY: finished
0626 19:57:47.674632 (+ 175us) log.cc:844] Serialized 29 byte log entry
0626 19:57:47.674801 (+ 169us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:47.674872 (+ 71us) write_op.cc:454] Released schema lock
0626 19:57:47.674909 (+ 37us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":205448,"reported_tablets":76,"child_traces":[["op",{"apply.queue_time_us":588,"apply.run_cpu_time_us":1344,"apply.run_wall_time_us":1343,"num_ops":2,"prepare.queue_time_us":17318,"prepare.run_cpu_time_us":2111,"prepare.run_wall_time_us":2111,"raft.queue_time_us":1196,"raft.run_cpu_time_us":639,"raft.run_wall_time_us":639,"replication_time_us":8892}]]}
I20250626 19:57:48.866395 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet 9172cb6576b240f6b25c3b1267618e6c with delete_type TABLET_DATA_DELETED (replaced by tablet 290232b0933b451c9d7744165bd72796) from {username='slave'} at 127.0.0.1:41356
W20250626 19:57:48.867360 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet f6ec952321924098a142f1438a283aab because the tablet was not found. No further retry: Not found: Tablet not found: f6ec952321924098a142f1438a283aab
W20250626 19:57:48.869833 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet fc526a05990446a3b3bc973e79511476 because the tablet was not found. No further retry: Not found: Tablet not found: fc526a05990446a3b3bc973e79511476
I20250626 19:57:48.866405 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 65af5ae255d74143976392bccc50f3ba with delete_type TABLET_DATA_DELETED (replaced by tablet a8ef6f024d0642d4a6dd67d646787868) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.872785 5937 ts_tablet_manager.cc:1905] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:48.874195 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4aee96ad3454d65b974c867605c4cdc" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
W20250626 19:57:48.875391 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 54e68eba747e4a2a947e84e53a444d08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54e68eba747e4a2a947e84e53a444d08 already in progress: creating tablet
W20250626 19:57:48.876061 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 3cc96a3e9c174cbca0196c5a1b27455c because the tablet was not found. No further retry: Not found: Tablet not found: 3cc96a3e9c174cbca0196c5a1b27455c
W20250626 19:57:48.876688 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet f807cf54bbb146ff989762c18410e6bb because the tablet was not found. No further retry: Not found: Tablet not found: f807cf54bbb146ff989762c18410e6bb
W20250626 19:57:48.877920 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 65af5ae255d74143976392bccc50f3ba because the tablet was not found. No further retry: Not found: Tablet not found: 65af5ae255d74143976392bccc50f3ba
W20250626 19:57:48.879361 4383 catalog_manager.cc:4908] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 1feb6785d0c7459fbc6898b0c9d3e38c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1feb6785d0c7459fbc6898b0c9d3e38c already in progress: creating tablet
I20250626 19:57:48.883356 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet f6ec952321924098a142f1438a283aab with delete_type TABLET_DATA_DELETED (replaced by tablet 2d0a30ee16d043f9837dce612dae670a) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.883471 6118 raft_consensus.cc:491] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:48.882936 5366 raft_consensus.cc:357] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:48.884243 5366 raft_consensus.cc:383] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:48.883975 6118 raft_consensus.cc:513] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:48.886015 6118 leader_election.cc:290] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:48.890136 6144 tablet.cc:1722] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:48.891322 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet fc526a05990446a3b3bc973e79511476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.895016 6154 tablet.cc:1722] T 0f834922c9de46dca48c47748913b11a P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:48.902050 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.904739 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 1feb6785d0c7459fbc6898b0c9d3e38c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.910506 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f319c748374b7cbbf037ef55aecd1b with delete_type TABLET_DATA_DELETED (replaced by tablet 15aade4064d742488ce29145185551a7) from {username='slave'} at 127.0.0.1:41356
W20250626 19:57:48.910620 4755 leader_election.cc:343] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:48.911175 4755 leader_election.cc:304] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:48.911933 6064 raft_consensus.cc:2747] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:57:48.912813 5965 ts_tablet_manager.cc:1918] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:48.913167 5965 log.cc:1199] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/3143181fae16411888d74f9152f3d5cc
I20250626 19:57:48.915482 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5c4cd6c2f64877a9e348dedb9820a3" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:48.873200 4672 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4aee96ad3454d65b974c867605c4cdc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:48.925223 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5c4cd6c2f64877a9e348dedb9820a3" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:48.932482 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 54e68eba747e4a2a947e84e53a444d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:48.935798 6155 tablet.cc:1722] T 465ed0fe56fb4960a58e92b9b3d594e6 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:48.884608 5366 raft_consensus.cc:738] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:48.943413 5366 consensus_queue.cc:260] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:48.854671 5353 raft_consensus.cc:357] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:48.947249 5353 raft_consensus.cc:383] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250626 19:57:48.947432 4393 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 70) took 3002 ms (client timeout 1000 ms). Trace:
I20250626 19:57:48.947594 5353 raft_consensus.cc:738] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:48.948081 5366 ts_tablet_manager.cc:1428] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.119s user 0.005s sys 0.000s
I20250626 19:57:48.948927 5366 tablet_bootstrap.cc:492] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
W20250626 19:57:48.947824 4393 rpcz_store.cc:269] 0626 19:57:45.945135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:45.965969 (+ 20834us) service_pool.cc:225] Handling call
0626 19:57:48.947360 (+2981391us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:47.290091 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:47.290532 (+ 441us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:47.290562 (+ 30us) write_op.cc:435] Acquired schema lock
0626 19:57:47.290578 (+ 16us) tablet.cc:660] Decoding operations
0626 19:57:47.290934 (+ 356us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:47.290983 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:47.291012 (+ 29us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:47.291339 (+ 327us) tablet.cc:699] Row locks acquired
0626 19:57:47.291354 (+ 15us) write_op.cc:260] PREPARE: finished
0626 19:57:47.291477 (+ 123us) write_op.cc:270] Start()
0626 19:57:47.291569 (+ 92us) write_op.cc:276] Timestamp: P: 1750967867291537 usec, L: 0
0626 19:57:47.291588 (+ 19us) op_driver.cc:348] REPLICATION: starting
0626 19:57:47.292091 (+ 503us) log.cc:844] Serialized 907 byte log entry
0626 19:57:47.299448 (+ 7357us) op_driver.cc:464] REPLICATION: finished
0626 19:57:47.299704 (+ 256us) write_op.cc:301] APPLY: starting
0626 19:57:47.299760 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:47.299863 (+ 103us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:47.299948 (+ 85us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:47.300176 (+ 228us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:47.300305 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:47.300348 (+ 43us) write_op.cc:312] APPLY: finished
0626 19:57:47.300510 (+ 162us) log.cc:844] Serialized 29 byte log entry
0626 19:57:47.300657 (+ 147us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:47.300722 (+ 65us) write_op.cc:454] Released schema lock
0626 19:57:47.300756 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:47.300961 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:47.300984 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2509,"reported_tablets":74,"child_traces":[["op",{"apply.queue_time_us":95,"apply.run_cpu_time_us":1236,"apply.run_wall_time_us":1236,"num_ops":2,"prepare.queue_time_us":145,"prepare.run_cpu_time_us":2307,"prepare.run_wall_time_us":2306,"raft.queue_time_us":1035,"raft.run_cpu_time_us":638,"raft.run_wall_time_us":637,"replication_time_us":7757,"spinlock_wait_cycles":1920}]]}
W20250626 19:57:48.959180 4756 leader_election.cc:343] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:48.971403 5353 consensus_queue.cc:260] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:48.973677 4393 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_35dcbe7e914846d8b6b52efa647c4d98" } modify_external_catalogs: true
I20250626 19:57:48.976022 4393 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_35dcbe7e914846d8b6b52efa647c4d98" } modify_external_catalogs: true
I20250626 19:57:48.977285 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4aee96ad3454d65b974c867605c4cdc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:48.975198 5965 ts_tablet_manager.cc:1939] T 3143181fae16411888d74f9152f3d5cc P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:48.980424 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 3143181fae16411888d74f9152f3d5cc (table default.Table_ad7eb0792b7444b2891bf198ef117218 [id=8ae02648a2e946c18cbb5f2d52ce375a]) successfully deleted
I20250626 19:57:48.980818 5965 tablet_replica.cc:331] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
W20250626 19:57:48.981261 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet e7c352224d0a46c6a922bf0a0d765f17 because the tablet was not found. No further retry: Not found: Tablet not found: e7c352224d0a46c6a922bf0a0d765f17
I20250626 19:57:48.982102 4912 leader_election.cc:304] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:48.977689 5353 ts_tablet_manager.cc:1428] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.125s user 0.007s sys 0.000s
I20250626 19:57:48.982900 6048 raft_consensus.cc:2747] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:48.983557 5353 tablet_bootstrap.cc:492] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:48.984279 6119 raft_consensus.cc:491] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:48.984794 6119 raft_consensus.cc:513] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:48.986819 6119 leader_election.cc:290] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:48.989262 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "075dbe46bab14d40a8008ce5c24bcf38" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:48.990267 4912 leader_election.cc:343] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:48.990774 4912 leader_election.cc:304] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:48.991503 6048 raft_consensus.cc:2747] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:48.993448 5965 raft_consensus.cc:2241] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:48.993909 5965 raft_consensus.cc:2270] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:48.995675 5965 ts_tablet_manager.cc:1905] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:48.997747 4912 leader_election.cc:343] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:49.025454 4391 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_6dc13f5ceb674e40919de2ca36d9427a" } modify_external_catalogs: true
I20250626 19:57:49.031984 5366 tablet_bootstrap.cc:654] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:49.034936 6143 tablet.cc:1722] T e0bcbb044d1549c88e5b83eeb5c709af P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:49.042838 5937 ts_tablet_manager.cc:1918] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:49.043368 5937 log.cc:1199] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/e7c352224d0a46c6a922bf0a0d765f17
I20250626 19:57:49.044131 5937 ts_tablet_manager.cc:1939] T e7c352224d0a46c6a922bf0a0d765f17 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:49.045486 5936 ts_tablet_manager.cc:1918] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:49.045876 5936 log.cc:1199] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/3143181fae16411888d74f9152f3d5cc
I20250626 19:57:49.046571 5936 ts_tablet_manager.cc:1939] T 3143181fae16411888d74f9152f3d5cc P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
W20250626 19:57:49.055310 4383 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 34b2919e5f534f4295336a4384be1f64 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 34b2919e5f534f4295336a4384be1f64 consensus state has no leader
W20250626 19:57:49.058741 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 61015.2us
I20250626 19:57:49.060925 5353 tablet_bootstrap.cc:654] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:49.062661 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 75135.3us
I20250626 19:57:49.087563 6162 tablet.cc:1722] T 465ed0fe56fb4960a58e92b9b3d594e6 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:49.089694 5353 tablet_bootstrap.cc:492] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:49.090158 5353 ts_tablet_manager.cc:1397] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.107s user 0.016s sys 0.004s
I20250626 19:57:49.092456 5353 raft_consensus.cc:357] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.093243 5353 raft_consensus.cc:383] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.093554 5353 raft_consensus.cc:738] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:49.094205 5353 consensus_queue.cc:260] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.096951 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "075dbe46bab14d40a8008ce5c24bcf38" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:49.098995 5353 ts_tablet_manager.cc:1428] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250626 19:57:49.099872 5353 tablet_bootstrap.cc:492] T 83f0431638144710b598b3f87d9a576e P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:49.104576 5366 tablet_bootstrap.cc:492] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:49.104986 5366 ts_tablet_manager.cc:1397] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.156s user 0.010s sys 0.001s
I20250626 19:57:49.106336 5924 raft_consensus.cc:491] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:49.106945 5924 raft_consensus.cc:513] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:49.109169 5924 leader_election.cc:290] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:49.110138 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fafbf98e7f4544b6ad96c1c5ab6c82bc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:49.110734 4816 raft_consensus.cc:2466] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:49.111294 5366 raft_consensus.cc:357] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.111922 5366 raft_consensus.cc:383] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.111979 4911 leader_election.cc:304] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:49.112151 5366 raft_consensus.cc:738] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:49.112808 5366 consensus_queue.cc:260] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.113983 6080 raft_consensus.cc:2802] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:49.114264 5366 ts_tablet_manager.cc:1428] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20250626 19:57:49.114305 6080 raft_consensus.cc:491] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:49.114584 6080 raft_consensus.cc:3058] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:49.117784 5353 tablet_bootstrap.cc:654] T 83f0431638144710b598b3f87d9a576e P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:49.135914 6080 raft_consensus.cc:513] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:49.139137 6080 leader_election.cc:290] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:49.082100 4755 leader_election.cc:343] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:49.141153 4755 leader_election.cc:304] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:49.141937 6122 raft_consensus.cc:2747] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:49.142079 4755 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 60043.7us
I20250626 19:57:49.142493 6157 tablet.cc:1722] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250626 19:57:49.143450 4755 leader_election.cc:343] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:49.156874 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fafbf98e7f4544b6ad96c1c5ab6c82bc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:49.157454 4819 raft_consensus.cc:3058] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:49.164454 4819 raft_consensus.cc:2466] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:49.165681 4911 leader_election.cc:304] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:49.166778 5924 raft_consensus.cc:2802] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:49.157117 5366 tablet_bootstrap.cc:492] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:49.167434 5924 raft_consensus.cc:695] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:49.168215 5924 consensus_queue.cc:237] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
W20250626 19:57:49.176409 4611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 210937us
W20250626 19:57:49.178903 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 117640us
W20250626 19:57:49.180413 4912 leader_election.cc:343] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:49.180910 4912 leader_election.cc:304] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:49.196177 5965 ts_tablet_manager.cc:1918] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:49.196663 5965 log.cc:1199] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/e94a73b50b564a97bbee0f5015bb8805
I20250626 19:57:49.197538 5410 tablet_bootstrap.cc:492] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:49.197435 5965 ts_tablet_manager.cc:1939] T e94a73b50b564a97bbee0f5015bb8805 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:49.197937 5410 ts_tablet_manager.cc:1397] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.725s user 0.007s sys 0.011s
W20250626 19:57:49.199278 4611 leader_election.cc:343] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:49.200775 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 136457us
I20250626 19:57:49.199743 4611 leader_election.cc:304] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:49.209219 6046 raft_consensus.cc:2747] T 83f0431638144710b598b3f87d9a576e P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:49.210992 5410 raft_consensus.cc:357] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.211656 5410 raft_consensus.cc:383] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.211907 5410 raft_consensus.cc:738] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:49.212462 5410 consensus_queue.cc:260] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.214030 5410 ts_tablet_manager.cc:1428] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250626 19:57:49.214532 6147 tablet.cc:1722] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:49.216492 6046 consensus_queue.cc:1035] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:49.217504 5353 tablet_bootstrap.cc:492] T 83f0431638144710b598b3f87d9a576e P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:49.217952 5353 ts_tablet_manager.cc:1397] T 83f0431638144710b598b3f87d9a576e P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.118s user 0.019s sys 0.013s
I20250626 19:57:49.221661 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet e7c352224d0a46c6a922bf0a0d765f17 (table default.Table_822bad36b81c42ac9101bcab4aaca9a4 [id=90440da2b42248e98586aa690f9199bd]) successfully deleted
W20250626 19:57:49.222509 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet f6ec952321924098a142f1438a283aab because the tablet was not found. No further retry: Not found: Tablet not found: f6ec952321924098a142f1438a283aab
W20250626 19:57:49.223107 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet cb34ffcd4da04e32a1220b3f298ec22d because the tablet was not found. No further retry: Not found: Tablet not found: cb34ffcd4da04e32a1220b3f298ec22d
W20250626 19:57:49.223664 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet e94a73b50b564a97bbee0f5015bb8805 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e94a73b50b564a97bbee0f5015bb8805 already in progress: creating tablet
W20250626 19:57:49.224210 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 36997dab0ab3430bba63a90805759b30 because the tablet was not found. No further retry: Not found: Tablet not found: 36997dab0ab3430bba63a90805759b30
W20250626 19:57:49.224751 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet fc526a05990446a3b3bc973e79511476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc526a05990446a3b3bc973e79511476 already in progress: creating tablet
W20250626 19:57:49.225287 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 4c0338c05a974e0f96865398722c5e8f because the tablet was not found. No further retry: Not found: Tablet not found: 4c0338c05a974e0f96865398722c5e8f
W20250626 19:57:49.225832 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet f807cf54bbb146ff989762c18410e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f807cf54bbb146ff989762c18410e6bb already in progress: creating tablet
W20250626 19:57:49.226377 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 65af5ae255d74143976392bccc50f3ba because the tablet was not found. No further retry: Not found: Tablet not found: 65af5ae255d74143976392bccc50f3ba
W20250626 19:57:49.227160 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 9172cb6576b240f6b25c3b1267618e6c because the tablet was not found. No further retry: Not found: Tablet not found: 9172cb6576b240f6b25c3b1267618e6c
W20250626 19:57:49.227779 4384 catalog_manager.cc:4895] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 609d36d46bb84017b2a3a7ad48bfccfc because the tablet was not found. No further retry: Not found: Tablet not found: 609d36d46bb84017b2a3a7ad48bfccfc
W20250626 19:57:49.228399 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 56f319c748374b7cbbf037ef55aecd1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56f319c748374b7cbbf037ef55aecd1b already in progress: creating tablet
W20250626 19:57:49.231974 4384 catalog_manager.cc:4908] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): delete failed for tablet 1feb6785d0c7459fbc6898b0c9d3e38c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1feb6785d0c7459fbc6898b0c9d3e38c already in progress: creating tablet
I20250626 19:57:49.237937 6048 raft_consensus.cc:2747] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:57:49.239252 5366 tablet_bootstrap.cc:654] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:49.238933 5353 raft_consensus.cc:357] T 83f0431638144710b598b3f87d9a576e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:49.239115 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 58763.9us
I20250626 19:57:49.239599 5353 raft_consensus.cc:383] T 83f0431638144710b598b3f87d9a576e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.239886 5353 raft_consensus.cc:738] T 83f0431638144710b598b3f87d9a576e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
W20250626 19:57:49.240178 4756 leader_election.cc:343] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:49.240594 4756 leader_election.cc:304] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:49.240500 5353 consensus_queue.cc:260] T 83f0431638144710b598b3f87d9a576e P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.241243 6121 raft_consensus.cc:2747] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:49.242013 5353 ts_tablet_manager.cc:1428] T 83f0431638144710b598b3f87d9a576e P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20250626 19:57:49.242834 5353 tablet_bootstrap.cc:492] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:49.243005 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet e94a73b50b564a97bbee0f5015bb8805 (table default.Table_fbcc45f0e38f4ed184900c0f58067165 [id=10f3db9788ba4edda2aab300e7313abb]) successfully deleted
W20250626 19:57:49.244688 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 9172cb6576b240f6b25c3b1267618e6c because the tablet was not found. No further retry: Not found: Tablet not found: 9172cb6576b240f6b25c3b1267618e6c
W20250626 19:57:49.245287 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 65af5ae255d74143976392bccc50f3ba because the tablet was not found. No further retry: Not found: Tablet not found: 65af5ae255d74143976392bccc50f3ba
W20250626 19:57:49.245775 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet f6ec952321924098a142f1438a283aab because the tablet was not found. No further retry: Not found: Tablet not found: f6ec952321924098a142f1438a283aab
W20250626 19:57:49.246274 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet fc526a05990446a3b3bc973e79511476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc526a05990446a3b3bc973e79511476 already in progress: creating tablet
W20250626 19:57:49.247061 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet f807cf54bbb146ff989762c18410e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f807cf54bbb146ff989762c18410e6bb already in progress: creating tablet
I20250626 19:57:49.253901 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 3143181fae16411888d74f9152f3d5cc (table default.Table_ad7eb0792b7444b2891bf198ef117218 [id=8ae02648a2e946c18cbb5f2d52ce375a]) successfully deleted
W20250626 19:57:49.256156 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 1feb6785d0c7459fbc6898b0c9d3e38c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1feb6785d0c7459fbc6898b0c9d3e38c already in progress: creating tablet
W20250626 19:57:49.259240 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 56f319c748374b7cbbf037ef55aecd1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56f319c748374b7cbbf037ef55aecd1b already in progress: creating tablet
W20250626 19:57:49.259848 4385 catalog_manager.cc:4908] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 54e68eba747e4a2a947e84e53a444d08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54e68eba747e4a2a947e84e53a444d08 already in progress: creating tablet
I20250626 19:57:49.261085 5353 tablet_bootstrap.cc:654] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:49.281483 5366 tablet_bootstrap.cc:492] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:49.281908 5366 ts_tablet_manager.cc:1397] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.125s user 0.003s sys 0.007s
I20250626 19:57:49.290071 5353 tablet_bootstrap.cc:492] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:49.290642 5353 ts_tablet_manager.cc:1397] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.005s
I20250626 19:57:49.294272 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fafbf98e7f4544b6ad96c1c5ab6c82bc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:49.295179 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fafbf98e7f4544b6ad96c1c5ab6c82bc" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:49.301327 5353 raft_consensus.cc:357] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.301968 5353 raft_consensus.cc:383] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.302227 5353 raft_consensus.cc:738] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:49.304937 5366 raft_consensus.cc:357] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.306079 6084 raft_consensus.cc:491] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:49.307119 5353 consensus_queue.cc:260] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.306604 6084 raft_consensus.cc:513] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.312165 6084 leader_election.cc:290] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:49.305559 5366 raft_consensus.cc:383] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.315035 5366 raft_consensus.cc:738] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:49.315763 5366 consensus_queue.cc:260] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.317739 4655 tablet_service.cc:1515] Processing DeleteTablet for tablet 3143181fae16411888d74f9152f3d5cc with delete_type TABLET_DATA_DELETED (replaced by tablet 08045a939eed4ec69b49d68b555bdf80) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:49.320946 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:49.324076 5353 ts_tablet_manager.cc:1428] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.033s user 0.002s sys 0.004s
I20250626 19:57:49.325465 6122 raft_consensus.cc:491] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:49.325919 6122 raft_consensus.cc:513] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.327772 6122 leader_election.cc:290] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:49.329164 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "159a6bf746e14479b3f3584640b63098" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:49.330904 4755 leader_election.cc:343] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:49.344695 5366 ts_tablet_manager.cc:1428] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.063s user 0.003s sys 0.004s
I20250626 19:57:49.345641 5366 tablet_bootstrap.cc:492] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:49.353052 5353 tablet_bootstrap.cc:492] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:49.355842 5026 heartbeater.cc:344] Connected to a master server at 127.4.46.190:33823
W20250626 19:57:49.374308 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 64097us
W20250626 19:57:49.379170 4912 leader_election.cc:343] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:49.382894 6103 log.cc:927] Time spent T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: Append to log took a long time: real 0.052s user 0.012s sys 0.003s
I20250626 19:57:49.358062 5353 tablet_bootstrap.cc:654] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:49.385264 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c5aba053ca4535ad2376b2066cfe47" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:49.385972 4816 raft_consensus.cc:2466] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:49.403093 5937 tablet_replica.cc:331] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:49.405619 5410 tablet_bootstrap.cc:492] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
W20250626 19:57:49.414533 4912 leader_election.cc:343] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:49.428658 6070 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 15327375340619623380) tablet 00000000000000000000000000000000 cur holder 0x7b5800072000
I20250626 19:57:49.448174 5937 raft_consensus.cc:2241] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:49.448886 5937 raft_consensus.cc:2270] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:49.460464 5937 ts_tablet_manager.cc:1905] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:49.463452 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 45414.8us
I20250626 19:57:49.473623 4609 leader_election.cc:304] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:49.476262 6048 raft_consensus.cc:491] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:49.476810 6048 raft_consensus.cc:513] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.478413 6085 raft_consensus.cc:2802] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:49.478910 6085 raft_consensus.cc:491] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:49.479399 6085 raft_consensus.cc:3058] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:49.480429 6048 leader_election.cc:290] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:49.480659 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c5aba053ca4535ad2376b2066cfe47" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:49.481194 4816 raft_consensus.cc:2466] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:49.481382 4396 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 188) took 2188 ms. Trace:
I20250626 19:57:49.483470 4911 leader_election.cc:304] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:49.481675 4396 rpcz_store.cc:276] 0626 19:57:47.293441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:48.796599 (+1503158us) service_pool.cc:225] Handling call
0626 19:57:48.812858 (+ 16259us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe791556d36 00007fe78f2736a4 00007fe78f273632 00007fe78f273561 00007fe78f2734dc 00007fe78f272792 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:48.846899 (+ 34041us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:48.848186 (+ 1287us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:49.007934 (+159748us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:49.481010 (+473076us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:49.481317 (+ 307us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:48.868112 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:48.868557 (+ 445us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:48.868583 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:48.868600 (+ 17us) tablet.cc:660] Decoding operations
0626 19:57:48.868906 (+ 306us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:48.868949 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:48.868976 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:48.869369 (+ 393us) tablet.cc:699] Row locks acquired
0626 19:57:48.869394 (+ 25us) write_op.cc:260] PREPARE: finished
0626 19:57:48.869519 (+ 125us) write_op.cc:270] Start()
0626 19:57:48.869616 (+ 97us) write_op.cc:276] Timestamp: P: 1750967868869595 usec, L: 0
0626 19:57:48.869633 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:48.870153 (+ 520us) log.cc:844] Serialized 433 byte log entry
0626 19:57:48.999751 (+129598us) op_driver.cc:464] REPLICATION: finished
0626 19:57:49.004192 (+ 4441us) write_op.cc:301] APPLY: starting
0626 19:57:49.004244 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:49.004359 (+ 115us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:49.004444 (+ 85us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:49.006409 (+ 1965us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:49.006547 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:49.006593 (+ 46us) write_op.cc:312] APPLY: finished
0626 19:57:49.006780 (+ 187us) log.cc:844] Serialized 29 byte log entry
0626 19:57:49.007339 (+ 559us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.007415 (+ 76us) write_op.cc:454] Released schema lock
0626 19:57:49.007450 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:49.007666 (+ 216us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.007703 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":34782080,"child_traces":[["op",{"apply.queue_time_us":4316,"apply.run_cpu_time_us":1520,"apply.run_wall_time_us":3440,"num_ops":2,"prepare.queue_time_us":108,"prepare.run_cpu_time_us":2174,"prepare.run_wall_time_us":3227,"replication_time_us":130047,"thread_start_us":3398,"threads_started":1}]]}
I20250626 19:57:49.490964 6080 raft_consensus.cc:2802] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:49.491333 6080 raft_consensus.cc:491] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:49.491631 6080 raft_consensus.cc:3058] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:49.493044 6046 raft_consensus.cc:491] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:49.493531 6046 raft_consensus.cc:513] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.496779 6046 leader_election.cc:290] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:49.500533 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c5aba053ca4535ad2376b2066cfe47" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:49.503127 4978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "159a6bf746e14479b3f3584640b63098" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:49.521308 4958 tablet_service.cc:1468] Processing CreateTablet for tablet 186597de5e9e4c7a91d33f59f88b40c0 (DEFAULT_TABLE table=default.Table_1e3c7d9cf5334791b285859e997796ee [id=c7b62da72835489dbd314a879f7a25d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:49.522442 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 186597de5e9e4c7a91d33f59f88b40c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:49.529412 4655 tablet_service.cc:1468] Processing CreateTablet for tablet 186597de5e9e4c7a91d33f59f88b40c0 (DEFAULT_TABLE table=default.Table_1e3c7d9cf5334791b285859e997796ee [id=c7b62da72835489dbd314a879f7a25d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:49.530053 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:49.530383 4655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 186597de5e9e4c7a91d33f59f88b40c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:49.530611 5353 tablet_bootstrap.cc:492] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:49.531047 5353 ts_tablet_manager.cc:1397] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.206s user 0.020s sys 0.022s
I20250626 19:57:49.533229 5353 raft_consensus.cc:357] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.533849 5353 raft_consensus.cc:383] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.534082 5353 raft_consensus.cc:738] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:49.540042 5410 tablet_bootstrap.cc:654] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:49.534627 5353 consensus_queue.cc:260] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.545049 4796 tablet_service.cc:1468] Processing CreateTablet for tablet 186597de5e9e4c7a91d33f59f88b40c0 (DEFAULT_TABLE table=default.Table_1e3c7d9cf5334791b285859e997796ee [id=c7b62da72835489dbd314a879f7a25d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:49.546119 4796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 186597de5e9e4c7a91d33f59f88b40c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:49.524521 5937 ts_tablet_manager.cc:1918] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:49.547188 5937 log.cc:1199] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/3143181fae16411888d74f9152f3d5cc
I20250626 19:57:49.547904 5937 ts_tablet_manager.cc:1939] T 3143181fae16411888d74f9152f3d5cc P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:49.557593 6080 raft_consensus.cc:513] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.559041 4979 raft_consensus.cc:2391] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:49.559614 6085 raft_consensus.cc:513] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.560894 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c5aba053ca4535ad2376b2066cfe47" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:49.561502 4816 raft_consensus.cc:3058] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:49.561553 6085 leader_election.cc:290] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:49.562503 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c5aba053ca4535ad2376b2066cfe47" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:49.563230 4670 raft_consensus.cc:2391] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:49.577258 4795 tablet_service.cc:1468] Processing CreateTablet for tablet 15aade4064d742488ce29145185551a7 (DEFAULT_TABLE table=default.Table_bf54a23958c14a1699f712665f29c0ad [id=a1cfdad9d9584a04be9e39bb35c60fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:49.578326 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15aade4064d742488ce29145185551a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:49.578828 5353 ts_tablet_manager.cc:1428] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20250626 19:57:49.586253 5353 tablet_bootstrap.cc:492] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:49.589157 4816 raft_consensus.cc:2466] T d7c5aba053ca4535ad2376b2066cfe47 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:49.597498 5353 tablet_bootstrap.cc:654] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:49.602591 5836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59366 (request call id 0) took 6050 ms. Trace:
I20250626 19:57:49.604115 6076 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ac80df82774b9b87c0be076b5637af. Latest consensus state: current_term: 6 leader_uuid: "62ac80df82774b9b87c0be076b5637af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:49.604799 6076 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:49.606539 6141 raft_consensus.cc:491] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 62ac80df82774b9b87c0be076b5637af)
I20250626 19:57:49.607115 6141 raft_consensus.cc:513] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:49.617210 5353 tablet_bootstrap.cc:492] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:49.617609 5353 ts_tablet_manager.cc:1397] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.000s
I20250626 19:57:49.620013 5353 raft_consensus.cc:357] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:49.620640 5353 raft_consensus.cc:383] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.620870 5353 raft_consensus.cc:738] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:49.621417 5353 consensus_queue.cc:260] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:49.515723 4394 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet c990e4eca5cf4a8e8295e9599cf5be29
I20250626 19:57:49.633981 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet c990e4eca5cf4a8e8295e9599cf5be29 with delete_type TABLET_DATA_DELETED (replaced by tablet c2ff9841bbbd429b83c123e1ebdabfea) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:49.634212 4394 catalog_manager.cc:6652] ReplaceTablet: tablet c990e4eca5cf4a8e8295e9599cf5be29 deleted and replaced by tablet c2ff9841bbbd429b83c123e1ebdabfea
I20250626 19:57:49.634671 5965 tablet_replica.cc:331] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:49.635036 4394 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34796 (request call id 189) took 1884 ms. Trace:
I20250626 19:57:49.635418 5965 raft_consensus.cc:2241] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:49.635879 5965 raft_consensus.cc:2270] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:49.516444 4395 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 187) took 2266 ms. Trace:
I20250626 19:57:49.516804 4391 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_6dc13f5ceb674e40919de2ca36d9427a" } modify_external_catalogs: true
I20250626 19:57:49.549054 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 3143181fae16411888d74f9152f3d5cc (table default.Table_ad7eb0792b7444b2891bf198ef117218 [id=8ae02648a2e946c18cbb5f2d52ce375a]) successfully deleted
I20250626 19:57:49.638695 5965 ts_tablet_manager.cc:1905] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:49.639055 4384 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 90065.9us
I20250626 19:57:49.640844 4649 tablet_service.cc:1468] Processing CreateTablet for tablet 15aade4064d742488ce29145185551a7 (DEFAULT_TABLE table=default.Table_bf54a23958c14a1699f712665f29c0ad [id=a1cfdad9d9584a04be9e39bb35c60fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:49.641814 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15aade4064d742488ce29145185551a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:49.642956 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet c990e4eca5cf4a8e8295e9599cf5be29 with delete_type TABLET_DATA_DELETED (replaced by tablet c2ff9841bbbd429b83c123e1ebdabfea) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:49.643572 5937 tablet_replica.cc:331] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:49.644294 5937 raft_consensus.cc:2241] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:49.644771 5937 raft_consensus.cc:2270] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:49.647413 5937 ts_tablet_manager.cc:1905] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:49.637600 4395 rpcz_store.cc:276] 0626 19:57:47.249776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:48.610087 (+1360311us) service_pool.cc:225] Handling call
0626 19:57:48.821808 (+211721us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe791556d36 00007fe78f2736a4 00007fe78f273632 00007fe78f273561 00007fe78f2734dc 00007fe78f272792 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:48.846120 (+ 24312us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:48.847765 (+ 1645us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:49.016231 (+168466us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:49.516100 (+499869us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:49.516380 (+ 280us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:48.849065 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:48.849524 (+ 459us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:48.849555 (+ 31us) write_op.cc:435] Acquired schema lock
0626 19:57:48.849569 (+ 14us) tablet.cc:660] Decoding operations
0626 19:57:48.849878 (+ 309us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:48.849920 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:48.849945 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:48.850148 (+ 203us) tablet.cc:699] Row locks acquired
0626 19:57:48.850166 (+ 18us) write_op.cc:260] PREPARE: finished
0626 19:57:48.850292 (+ 126us) write_op.cc:270] Start()
0626 19:57:48.850382 (+ 90us) write_op.cc:276] Timestamp: P: 1750967868850359 usec, L: 0
0626 19:57:48.850399 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:57:48.850963 (+ 564us) log.cc:844] Serialized 433 byte log entry
0626 19:57:48.999314 (+148351us) op_driver.cc:464] REPLICATION: finished
0626 19:57:49.003311 (+ 3997us) write_op.cc:301] APPLY: starting
0626 19:57:49.003368 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:49.004004 (+ 636us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:49.014930 (+ 10926us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:49.015373 (+ 443us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:49.015513 (+ 140us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:49.015556 (+ 43us) write_op.cc:312] APPLY: finished
0626 19:57:49.015748 (+ 192us) log.cc:844] Serialized 29 byte log entry
0626 19:57:49.015916 (+ 168us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.015981 (+ 65us) write_op.cc:454] Released schema lock
0626 19:57:49.016016 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:49.016738 (+ 722us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.016793 (+ 55us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":485215104,"child_traces":[["op",{"apply.queue_time_us":3845,"apply.run_cpu_time_us":1619,"apply.run_wall_time_us":13326,"num_ops":2,"prepare.queue_time_us":92,"prepare.run_cpu_time_us":2097,"prepare.run_wall_time_us":3017,"replication_time_us":148806}]]}
I20250626 19:57:49.658587 5353 ts_tablet_manager.cc:1428] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20250626 19:57:49.659929 6171 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "62ac80df82774b9b87c0be076b5637af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:49.660907 6171 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:57:49.663467 4396 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_d51d8fa1843f4e47a913158768b4689c" } new_table_name: "default.Table_b8d28650a552496b9582e1623ea694ff" modify_external_catalogs: true
I20250626 19:57:49.664865 6141 leader_election.cc:290] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:49.665335 5353 tablet_bootstrap.cc:492] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:49.668542 5937 ts_tablet_manager.cc:1918] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250626 19:57:49.668886 5937 log.cc:1199] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/c990e4eca5cf4a8e8295e9599cf5be29
I20250626 19:57:49.669587 5937 ts_tablet_manager.cc:1939] T c990e4eca5cf4a8e8295e9599cf5be29 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:49.675320 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]) successfully deleted
I20250626 19:57:49.684630 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 15aade4064d742488ce29145185551a7 (DEFAULT_TABLE table=default.Table_bf54a23958c14a1699f712665f29c0ad [id=a1cfdad9d9584a04be9e39bb35c60fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:49.685629 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15aade4064d742488ce29145185551a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:49.687091 4952 tablet_service.cc:1468] Processing CreateTablet for tablet 4631acfeee24418c8fc5f8a6416cbda9 (DEFAULT_TABLE table=default.Table_91248389db7e4324a3ddc4e5874b2666 [id=ffbbbd1dc6384b4a8aa92097ecb77f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:49.688028 4952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4631acfeee24418c8fc5f8a6416cbda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:49.692062 5410 tablet_bootstrap.cc:492] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:49.692487 5410 ts_tablet_manager.cc:1397] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.287s user 0.010s sys 0.000s
I20250626 19:57:49.694677 5410 raft_consensus.cc:357] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:49.695317 5410 raft_consensus.cc:383] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.695562 5410 raft_consensus.cc:738] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:49.696094 5410 consensus_queue.cc:260] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:49.697737 5410 ts_tablet_manager.cc:1428] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250626 19:57:49.698838 5410 tablet_bootstrap.cc:492] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:49.700907 5965 ts_tablet_manager.cc:1918] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250626 19:57:49.701292 5965 log.cc:1199] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/c990e4eca5cf4a8e8295e9599cf5be29
I20250626 19:57:49.702018 5965 ts_tablet_manager.cc:1939] T c990e4eca5cf4a8e8295e9599cf5be29 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:49.705104 5353 tablet_bootstrap.cc:654] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:49.708156 5410 tablet_bootstrap.cc:654] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:49.710516 6080 leader_election.cc:290] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:49.717502 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]) successfully deleted
I20250626 19:57:49.635433 4394 rpcz_store.cc:276] 0626 19:57:47.750843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:48.803342 (+1052499us) service_pool.cc:225] Handling call
0626 19:57:48.815067 (+ 11725us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72ba70 00007fe78f7afb1a 00007fe79145cfac 00007fe791559891 00007fe78f2943a7 00007fe78f294332 00007fe78f294261 00007fe78f2941dc 00007fe78f293492 00007fe78d56fdcc 00007fe78d56f206
0626 19:57:49.634976 (+819909us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:48.817836 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:48.818260 (+ 424us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:48.818292 (+ 32us) write_op.cc:435] Acquired schema lock
0626 19:57:48.818307 (+ 15us) tablet.cc:660] Decoding operations
0626 19:57:48.818644 (+ 337us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:48.818691 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:48.818716 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:48.819170 (+ 454us) tablet.cc:699] Row locks acquired
0626 19:57:48.819185 (+ 15us) write_op.cc:260] PREPARE: finished
0626 19:57:48.819304 (+ 119us) write_op.cc:270] Start()
0626 19:57:48.819389 (+ 85us) write_op.cc:276] Timestamp: P: 1750967868819361 usec, L: 0
0626 19:57:48.819403 (+ 14us) op_driver.cc:348] REPLICATION: starting
0626 19:57:48.819908 (+ 505us) log.cc:844] Serialized 659 byte log entry
0626 19:57:48.919694 (+ 99786us) op_driver.cc:464] REPLICATION: finished
0626 19:57:49.000160 (+ 80466us) write_op.cc:301] APPLY: starting
0626 19:57:49.000233 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:49.000384 (+ 151us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:49.000497 (+ 113us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:49.000980 (+ 483us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:49.001249 (+ 269us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:49.001314 (+ 65us) write_op.cc:312] APPLY: finished
0626 19:57:49.001561 (+ 247us) log.cc:844] Serialized 29 byte log entry
0626 19:57:49.001781 (+ 220us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.001871 (+ 90us) write_op.cc:454] Released schema lock
0626 19:57:49.001920 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:49.002928 (+ 1008us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.002969 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":24607104,"child_traces":[["op",{"apply.queue_time_us":80287,"apply.run_cpu_time_us":2016,"apply.run_wall_time_us":2014,"num_ops":2,"prepare.queue_time_us":95,"prepare.run_cpu_time_us":2084,"prepare.run_wall_time_us":2265,"replication_time_us":100193}]]}
I20250626 19:57:49.721349 4394 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34796:
table { table_name: "default.Table_bd7be9809f0d462380d65cb6ed3f8c1f" } new_table_name: "default.Table_1f95cb776ca24388979785af3b4e84c6" modify_external_catalogs: true
I20250626 19:57:49.603035 5836 rpcz_store.cc:276] 0626 19:57:43.552602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:43.552736 (+ 134us) service_pool.cc:225] Handling call
0626 19:57:48.054568 (+4501832us) raft_consensus.cc:1469] Updating replica for 139 ops
0626 19:57:48.184838 (+130270us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:48.184873 (+ 35us) write_op.cc:454] Released schema lock
0626 19:57:48.185815 (+ 942us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:48.185843 (+ 28us) write_op.cc:454] Released schema lock
0626 19:57:48.186554 (+ 711us) raft_consensus.cc:1532] Early marking committed up to index 85
0626 19:57:48.324313 (+137759us) raft_consensus.cc:1537] Triggering prepare for 139 ops
0626 19:57:48.324975 (+ 662us) write_op.cc:270] Start()
0626 19:57:48.325491 (+ 516us) write_op.cc:276] Timestamp: P: 1750967850499134 usec, L: 0
0626 19:57:48.326232 (+ 741us) write_op.cc:270] Start()
0626 19:57:48.326350 (+ 118us) write_op.cc:276] Timestamp: P: 1750967850547323 usec, L: 0
0626 19:57:48.326943 (+ 593us) write_op.cc:270] Start()
0626 19:57:48.327047 (+ 104us) write_op.cc:276] Timestamp: P: 1750967850637148 usec, L: 0
0626 19:57:48.327620 (+ 573us) write_op.cc:270] Start()
0626 19:57:48.327711 (+ 91us) write_op.cc:276] Timestamp: P: 1750967850641542 usec, L: 0
0626 19:57:48.328374 (+ 663us) write_op.cc:270] Start()
0626 19:57:48.328492 (+ 118us) write_op.cc:276] Timestamp: P: 1750967850701982 usec, L: 0
0626 19:57:48.329015 (+ 523us) write_op.cc:270] Start()
0626 19:57:48.329121 (+ 106us) write_op.cc:276] Timestamp: P: 1750967850770785 usec, L: 0
0626 19:57:48.329641 (+ 520us) write_op.cc:270] Start()
0626 19:57:48.329738 (+ 97us) write_op.cc:276] Timestamp: P: 1750967850775752 usec, L: 0
0626 19:57:48.330246 (+ 508us) write_op.cc:270] Start()
0626 19:57:48.330336 (+ 90us) write_op.cc:276] Timestamp: P: 1750967850816786 usec, L: 0
0626 19:57:48.380029 (+ 49693us) write_op.cc:270] Start()
0626 19:57:48.380134 (+ 105us) write_op.cc:276] Timestamp: P: 1750967850909208 usec, L: 0
0626 19:57:48.380740 (+ 606us) write_op.cc:270] Start()
0626 19:57:48.380821 (+ 81us) write_op.cc:276] Timestamp: P: 1750967850937822 usec, L: 0
0626 19:57:48.381406 (+ 585us) write_op.cc:270] Start()
0626 19:57:48.381504 (+ 98us) write_op.cc:276] Timestamp: P: 1750967850992303 usec, L: 0
0626 19:57:48.382148 (+ 644us) write_op.cc:270] Start()
0626 19:57:48.382282 (+ 134us) write_op.cc:276] Timestamp: P: 1750967851133423 usec, L: 0
0626 19:57:48.382943 (+ 661us) write_op.cc:270] Start()
0626 19:57:48.383052 (+ 109us) write_op.cc:276] Timestamp: P: 1750967851136972 usec, L: 0
0626 19:57:48.383613 (+ 561us) write_op.cc:270] Start()
0626 19:57:48.383705 (+ 92us) write_op.cc:276] Timestamp: P: 1750967851254221 usec, L: 0
0626 19:57:48.384268 (+ 563us) write_op.cc:270] Start()
0626 19:57:48.384364 (+ 96us) write_op.cc:276] Timestamp: P: 1750967851278786 usec, L: 0
0626 19:57:48.384905 (+ 541us) write_op.cc:270] Start()
0626 19:57:48.384997 (+ 92us) write_op.cc:276] Timestamp: P: 1750967851369783 usec, L: 0
0626 19:57:48.385602 (+ 605us) write_op.cc:270] Start()
0626 19:57:48.385699 (+ 97us) write_op.cc:276] Timestamp: P: 1750967851403598 usec, L: 0
0626 19:57:48.386233 (+ 534us) write_op.cc:270] Start()
0626 19:57:48.386329 (+ 96us) write_op.cc:276] Timestamp: P: 1750967851463365 usec, L: 0
0626 19:57:48.412475 (+ 26146us) write_op.cc:270] Start()
0626 19:57:48.412575 (+ 100us) write_op.cc:276] Timestamp: P: 1750967851497515 usec, L: 0
0626 19:57:48.413091 (+ 516us) write_op.cc:270] Start()
0626 19:57:48.413170 (+ 79us) write_op.cc:276] Timestamp: P: 1750967851654347 usec, L: 0
0626 19:57:48.419260 (+ 6090us) write_op.cc:270] Start()
0626 19:57:48.419373 (+ 113us) write_op.cc:276] Timestamp: P: 1750967851657824 usec, L: 0
0626 19:57:48.419966 (+ 593us) write_op.cc:270] Start()
0626 19:57:48.420183 (+ 217us) write_op.cc:276] Timestamp: P: 1750967851807053 usec, L: 0
0626 19:57:48.420659 (+ 476us) write_op.cc:270] Start()
0626 19:57:48.420732 (+ 73us) write_op.cc:276] Timestamp: P: 1750967851809732 usec, L: 0
0626 19:57:48.421163 (+ 431us) write_op.cc:270] Start()
0626 19:57:48.421282 (+ 119us) write_op.cc:276] Timestamp: P: 1750967851967259 usec, L: 0
0626 19:57:48.421754 (+ 472us) write_op.cc:270] Start()
0626 19:57:48.421842 (+ 88us) write_op.cc:276] Timestamp: P: 1750967851969603 usec, L: 0
0626 19:57:48.422272 (+ 430us) write_op.cc:270] Start()
0626 19:57:48.422343 (+ 71us) write_op.cc:276] Timestamp: P: 1750967852058669 usec, L: 0
0626 19:57:48.463524 (+ 41181us) write_op.cc:270] Start()
0626 19:57:48.463644 (+ 120us) write_op.cc:276] Timestamp: P: 1750967852060866 usec, L: 0
0626 19:57:48.464203 (+ 559us) write_op.cc:270] Start()
0626 19:57:48.464280 (+ 77us) write_op.cc:276] Timestamp: P: 1750967852083029 usec, L: 0
0626 19:57:48.464875 (+ 595us) write_op.cc:270] Start()
0626 19:57:48.464976 (+ 101us) write_op.cc:276] Timestamp: P: 1750967852095999 usec, L: 0
0626 19:57:48.465524 (+ 548us) write_op.cc:270] Start()
0626 19:57:48.465618 (+ 94us) write_op.cc:276] Timestamp: P: 1750967852390265 usec, L: 0
0626 19:57:48.466260 (+ 642us) write_op.cc:270] Start()
0626 19:57:48.466342 (+ 82us) write_op.cc:276] Timestamp: P: 1750967852441128 usec, L: 0
0626 19:57:48.492509 (+ 26167us) write_op.cc:270] Start()
0626 19:57:48.492608 (+ 99us) write_op.cc:276] Timestamp: P: 1750967852460903 usec, L: 0
0626 19:57:48.493248 (+ 640us) write_op.cc:270] Start()
0626 19:57:48.493336 (+ 88us) write_op.cc:276] Timestamp: P: 1750967852879591 usec, L: 0
0626 19:57:48.493810 (+ 474us) write_op.cc:270] Start()
0626 19:57:48.493888 (+ 78us) write_op.cc:276] Timestamp: P: 1750967852943657 usec, L: 0
0626 19:57:48.494316 (+ 428us) write_op.cc:270] Start()
0626 19:57:48.494388 (+ 72us) write_op.cc:276] Timestamp: P: 1750967853093218 usec, L: 0
0626 19:57:48.494891 (+ 503us) write_op.cc:270] Start()
0626 19:57:48.494967 (+ 76us) write_op.cc:276] Timestamp: P: 1750967853193530 usec, L: 0
0626 19:57:48.495414 (+ 447us) write_op.cc:270] Start()
0626 19:57:48.495504 (+ 90us) write_op.cc:276] Timestamp: P: 1750967853333896 usec, L: 0
0626 19:57:48.495964 (+ 460us) write_op.cc:270] Start()
0626 19:57:48.496044 (+ 80us) write_op.cc:276] Timestamp: P: 1750967853388630 usec, L: 0
0626 19:57:48.496483 (+ 439us) write_op.cc:270] Start()
0626 19:57:48.496564 (+ 81us) write_op.cc:276] Timestamp: P: 1750967853577617 usec, L: 0
0626 19:57:48.497001 (+ 437us) write_op.cc:270] Start()
0626 19:57:48.497077 (+ 76us) write_op.cc:276] Timestamp: P: 1750967853586671 usec, L: 0
0626 19:57:48.497542 (+ 465us) write_op.cc:270] Start()
0626 19:57:48.497626 (+ 84us) write_op.cc:276] Timestamp: P: 1750967853602323 usec, L: 0
0626 19:57:48.498062 (+ 436us) write_op.cc:270] Start()
0626 19:57:48.498136 (+ 74us) write_op.cc:276] Timestamp: P: 1750967853656095 usec, L: 0
0626 19:57:48.498566 (+ 430us) write_op.cc:270] Start()
0626 19:57:48.498636 (+ 70us) write_op.cc:276] Timestamp: P: 1750967853680535 usec, L: 0
0626 19:57:48.535363 (+ 36727us) write_op.cc:270] Start()
0626 19:57:48.535495 (+ 132us) write_op.cc:276] Timestamp: P: 1750967853754485 usec, L: 0
0626 19:57:48.536158 (+ 663us) write_op.cc:270] Start()
0626 19:57:48.536277 (+ 119us) write_op.cc:276] Timestamp: P: 1750967853882119 usec, L: 0
0626 19:57:48.536874 (+ 597us) write_op.cc:270] Start()
0626 19:57:48.536975 (+ 101us) write_op.cc:276] Timestamp: P: 1750967853921281 usec, L: 0
0626 19:57:48.559098 (+ 22123us) write_op.cc:270] Start()
0626 19:57:48.559191 (+ 93us) write_op.cc:276] Timestamp: P: 1750967853962724 usec, L: 0
0626 19:57:48.559689 (+ 498us) write_op.cc:270] Start()
0626 19:57:48.559839 (+ 150us) write_op.cc:276] Timestamp: P: 1750967854111185 usec, L: 0
0626 19:57:48.560347 (+ 508us) write_op.cc:270] Start()
0626 19:57:48.560426 (+ 79us) write_op.cc:276] Timestamp: P: 1750967854154239 usec, L: 0
0626 19:57:48.560888 (+ 462us) write_op.cc:270] Start()
0626 19:57:48.560959 (+ 71us) write_op.cc:276] Timestamp: P: 1750967854189867 usec, L: 0
0626 19:57:48.561377 (+ 418us) write_op.cc:270] Start()
0626 19:57:48.561459 (+ 82us) write_op.cc:276] Timestamp: P: 1750967854218375 usec, L: 0
0626 19:57:48.570624 (+ 9165us) write_op.cc:270] Start()
0626 19:57:48.570793 (+ 169us) write_op.cc:276] Timestamp: P: 1750967854237815 usec, L: 0
0626 19:57:48.581850 (+ 11057us) write_op.cc:270] Start()
0626 19:57:48.581992 (+ 142us) write_op.cc:276] Timestamp: P: 1750967854252953 usec, L: 0
0626 19:57:48.582644 (+ 652us) write_op.cc:270] Start()
0626 19:57:48.582726 (+ 82us) write_op.cc:276] Timestamp: P: 1750967854259899 usec, L: 0
0626 19:57:48.583218 (+ 492us) write_op.cc:270] Start()
0626 19:57:48.583297 (+ 79us) write_op.cc:276] Timestamp: P: 1750967854384770 usec, L: 0
0626 19:57:48.583784 (+ 487us) write_op.cc:270] Start()
0626 19:57:48.583865 (+ 81us) write_op.cc:276] Timestamp: P: 1750967854394856 usec, L: 0
0626 19:57:48.584334 (+ 469us) write_op.cc:270] Start()
0626 19:57:48.584428 (+ 94us) write_op.cc:276] Timestamp: P: 1750967854497608 usec, L: 0
0626 19:57:48.584999 (+ 571us) write_op.cc:270] Start()
0626 19:57:48.585074 (+ 75us) write_op.cc:276] Timestamp: P: 1750967854508715 usec, L: 0
0626 19:57:48.585537 (+ 463us) write_op.cc:270] Start()
0626 19:57:48.585624 (+ 87us) write_op.cc:276] Timestamp: P: 1750967854608233 usec, L: 0
0626 19:57:48.586104 (+ 480us) write_op.cc:270] Start()
0626 19:57:48.586184 (+ 80us) write_op.cc:276] Timestamp: P: 1750967854674761 usec, L: 0
0626 19:57:48.586687 (+ 503us) write_op.cc:270] Start()
0626 19:57:48.704701 (+118014us) write_op.cc:276] Timestamp: P: 1750967854700927 usec, L: 0
0626 19:57:48.706696 (+ 1995us) write_op.cc:270] Start()
0626 19:57:48.736301 (+ 29605us) write_op.cc:276] Timestamp: P: 1750967854780678 usec, L: 0
0626 19:57:48.736944 (+ 643us) write_op.cc:270] Start()
0626 19:57:48.737042 (+ 98us) write_op.cc:276] Timestamp: P: 1750967854808566 usec, L: 0
0626 19:57:48.737550 (+ 508us) write_op.cc:270] Start()
0626 19:57:48.737630 (+ 80us) write_op.cc:276] Timestamp: P: 1750967854821135 usec, L: 0
0626 19:57:48.738232 (+ 602us) write_op.cc:270] Start()
0626 19:57:48.738315 (+ 83us) write_op.cc:276] Timestamp: P: 1750967854941697 usec, L: 0
0626 19:57:48.767158 (+ 28843us) write_op.cc:270] Start()
0626 19:57:48.767286 (+ 128us) write_op.cc:276] Timestamp: P: 1750967855003083 usec, L: 0
0626 19:57:48.767986 (+ 700us) write_op.cc:270] Start()
0626 19:57:48.768197 (+ 211us) write_op.cc:276] Timestamp: P: 1750967855024389 usec, L: 0
0626 19:57:48.768893 (+ 696us) write_op.cc:270] Start()
0626 19:57:48.768992 (+ 99us) write_op.cc:276] Timestamp: P: 1750967855051736 usec, L: 0
0626 19:57:48.769563 (+ 571us) write_op.cc:270] Start()
0626 19:57:48.769675 (+ 112us) write_op.cc:276] Timestamp: P: 1750967855124800 usec, L: 0
0626 19:57:48.770212 (+ 537us) write_op.cc:270] Start()
0626 19:57:48.770308 (+ 96us) write_op.cc:276] Timestamp: P: 1750967855223000 usec, L: 0
0626 19:57:48.792617 (+ 22309us) write_op.cc:270] Start()
0626 19:57:48.792735 (+ 118us) write_op.cc:276] Timestamp: P: 1750967855297946 usec, L: 0
0626 19:57:48.793312 (+ 577us) write_op.cc:270] Start()
0626 19:57:48.793394 (+ 82us) write_op.cc:276] Timestamp: P: 1750967855417517 usec, L: 0
0626 19:57:48.793925 (+ 531us) write_op.cc:270] Start()
0626 19:57:48.794007 (+ 82us) write_op.cc:276] Timestamp: P: 1750967855641437 usec, L: 0
0626 19:57:48.794449 (+ 442us) write_op.cc:270] Start()
0626 19:57:48.794531 (+ 82us) write_op.cc:276] Timestamp: P: 1750967855745977 usec, L: 0
0626 19:57:48.795025 (+ 494us) write_op.cc:270] Start()
0626 19:57:48.795106 (+ 81us) write_op.cc:276] Timestamp: P: 1750967855964864 usec, L: 0
0626 19:57:48.870998 (+ 75892us) write_op.cc:270] Start()
0626 19:57:48.871109 (+ 111us) write_op.cc:276] Timestamp: P: 1750967856068901 usec, L: 0
0626 19:57:48.871686 (+ 577us) write_op.cc:270] Start()
0626 19:57:48.871767 (+ 81us) write_op.cc:276] Timestamp: P: 1750967856226530 usec, L: 0
0626 19:57:48.872207 (+ 440us) write_op.cc:270] Start()
0626 19:57:48.872296 (+ 89us) write_op.cc:276] Timestamp: P: 1750967856415444 usec, L: 0
0626 19:57:48.872734 (+ 438us) write_op.cc:270] Start()
0626 19:57:48.872805 (+ 71us) write_op.cc:276] Timestamp: P: 1750967856441355 usec, L: 0
0626 19:57:48.873236 (+ 431us) write_op.cc:270] Start()
0626 19:57:48.873306 (+ 70us) write_op.cc:276] Timestamp: P: 1750967856465617 usec, L: 0
0626 19:57:48.873761 (+ 455us) write_op.cc:270] Start()
0626 19:57:48.873836 (+ 75us) write_op.cc:276] Timestamp: P: 1750967856489977 usec, L: 0
0626 19:57:48.874259 (+ 423us) write_op.cc:270] Start()
0626 19:57:48.874338 (+ 79us) write_op.cc:276] Timestamp: P: 1750967856910710 usec, L: 0
0626 19:57:48.937458 (+ 63120us) write_op.cc:270] Start()
0626 19:57:48.937559 (+ 101us) write_op.cc:276] Timestamp: P: 1750967857063601 usec, L: 0
0626 19:57:48.938126 (+ 567us) write_op.cc:270] Start()
0626 19:57:48.938203 (+ 77us) write_op.cc:276] Timestamp: P: 1750967857065473 usec, L: 0
0626 19:57:48.938663 (+ 460us) write_op.cc:270] Start()
0626 19:57:48.938739 (+ 76us) write_op.cc:276] Timestamp: P: 1750967857068020 usec, L: 0
0626 19:57:48.939215 (+ 476us) write_op.cc:270] Start()
0626 19:57:48.939288 (+ 73us) write_op.cc:276] Timestamp: P: 1750967857069990 usec, L: 0
0626 19:57:48.939742 (+ 454us) write_op.cc:270] Start()
0626 19:57:48.939824 (+ 82us) write_op.cc:276] Timestamp: P: 1750967857133064 usec, L: 0
0626 19:57:48.963883 (+ 24059us) write_op.cc:270] Start()
0626 19:57:48.963998 (+ 115us) write_op.cc:276] Timestamp: P: 1750967857135060 usec, L: 0
0626 19:57:48.964579 (+ 581us) write_op.cc:270] Start()
0626 19:57:48.964663 (+ 84us) write_op.cc:276] Timestamp: P: 1750967857148042 usec, L: 0
0626 19:57:48.965141 (+ 478us) write_op.cc:270] Start()
0626 19:57:48.965229 (+ 88us) write_op.cc:276] Timestamp: P: 1750967857167042 usec, L: 0
0626 19:57:48.966916 (+ 1687us) write_op.cc:270] Start()
0626 19:57:48.967008 (+ 92us) write_op.cc:276] Timestamp: P: 1750967857256103 usec, L: 0
0626 19:57:48.967652 (+ 644us) write_op.cc:270] Start()
0626 19:57:48.967724 (+ 72us) write_op.cc:276] Timestamp: P: 1750967857372174 usec, L: 0
0626 19:57:48.968169 (+ 445us) write_op.cc:270] Start()
0626 19:57:48.968331 (+ 162us) write_op.cc:276] Timestamp: P: 1750967857430473 usec, L: 0
0626 19:57:48.968739 (+ 408us) write_op.cc:270] Start()
0626 19:57:48.968809 (+ 70us) write_op.cc:276] Timestamp: P: 1750967857432569 usec, L: 0
0626 19:57:48.969243 (+ 434us) write_op.cc:270] Start()
0626 19:57:48.969311 (+ 68us) write_op.cc:276] Timestamp: P: 1750967857436431 usec, L: 0
0626 19:57:48.969722 (+ 411us) write_op.cc:270] Start()
0626 19:57:48.969796 (+ 74us) write_op.cc:276] Timestamp: P: 1750967857440490 usec, L: 0
0626 19:57:48.970206 (+ 410us) write_op.cc:270] Start()
0626 19:57:48.970276 (+ 70us) write_op.cc:276] Timestamp: P: 1750967857492270 usec, L: 0
0626 19:57:48.970673 (+ 397us) write_op.cc:270] Start()
0626 19:57:49.034042 (+ 63369us) write_op.cc:276] Timestamp: P: 1750967857587815 usec, L: 0
0626 19:57:49.034933 (+ 891us) write_op.cc:270] Start()
0626 19:57:49.035032 (+ 99us) write_op.cc:276] Timestamp: P: 1750967857640545 usec, L: 0
0626 19:57:49.035564 (+ 532us) write_op.cc:270] Start()
0626 19:57:49.035654 (+ 90us) write_op.cc:276] Timestamp: P: 1750967857706523 usec, L: 0
0626 19:57:49.036119 (+ 465us) write_op.cc:270] Start()
0626 19:57:49.036191 (+ 72us) write_op.cc:276] Timestamp: P: 1750967857773938 usec, L: 0
0626 19:57:49.036632 (+ 441us) write_op.cc:270] Start()
0626 19:57:49.036711 (+ 79us) write_op.cc:276] Timestamp: P: 1750967857795514 usec, L: 0
0626 19:57:49.037144 (+ 433us) write_op.cc:270] Start()
0626 19:57:49.037211 (+ 67us) write_op.cc:276] Timestamp: P: 1750967857822224 usec, L: 0
0626 19:57:49.037629 (+ 418us) write_op.cc:270] Start()
0626 19:57:49.037697 (+ 68us) write_op.cc:276] Timestamp: P: 1750967857885822 usec, L: 0
0626 19:57:49.038087 (+ 390us) write_op.cc:270] Start()
0626 19:57:49.038147 (+ 60us) write_op.cc:276] Timestamp: P: 1750967857940646 usec, L: 0
0626 19:57:49.038537 (+ 390us) write_op.cc:270] Start()
0626 19:57:49.038614 (+ 77us) write_op.cc:276] Timestamp: P: 1750967857984874 usec, L: 0
0626 19:57:49.098282 (+ 59668us) write_op.cc:270] Start()
0626 19:57:49.098369 (+ 87us) write_op.cc:276] Timestamp: P: 1750967858325751 usec, L: 0
0626 19:57:49.098853 (+ 484us) write_op.cc:270] Start()
0626 19:57:49.098920 (+ 67us) write_op.cc:276] Timestamp: P: 1750967858355525 usec, L: 0
0626 19:57:49.099345 (+ 425us) write_op.cc:270] Start()
0626 19:57:49.099412 (+ 67us) write_op.cc:276] Timestamp: P: 1750967858560227 usec, L: 0
0626 19:57:49.099859 (+ 447us) write_op.cc:270] Start()
0626 19:57:49.099929 (+ 70us) write_op.cc:276] Timestamp: P: 1750967858901684 usec, L: 0
0626 19:57:49.100385 (+ 456us) write_op.cc:270] Start()
0626 19:57:49.100470 (+ 85us) write_op.cc:276] Timestamp: P: 1750967859214250 usec, L: 0
0626 19:57:49.100919 (+ 449us) write_op.cc:270] Start()
0626 19:57:49.100990 (+ 71us) write_op.cc:276] Timestamp: P: 1750967859216274 usec, L: 0
0626 19:57:49.101397 (+ 407us) write_op.cc:270] Start()
0626 19:57:49.101466 (+ 69us) write_op.cc:276] Timestamp: P: 1750967859366388 usec, L: 0
0626 19:57:49.102667 (+ 1201us) write_op.cc:270] Start()
0626 19:57:49.143034 (+ 40367us) write_op.cc:276] Timestamp: P: 1750967859436797 usec, L: 0
0626 19:57:49.143647 (+ 613us) write_op.cc:270] Start()
0626 19:57:49.143732 (+ 85us) write_op.cc:276] Timestamp: P: 1750967859623208 usec, L: 0
0626 19:57:49.144184 (+ 452us) write_op.cc:270] Start()
0626 19:57:49.144253 (+ 69us) write_op.cc:276] Timestamp: P: 1750967859626601 usec, L: 0
0626 19:57:49.163613 (+ 19360us) write_op.cc:270] Start()
0626 19:57:49.163697 (+ 84us) write_op.cc:276] Timestamp: P: 1750967859630902 usec, L: 0
0626 19:57:49.164142 (+ 445us) write_op.cc:270] Start()
0626 19:57:49.164220 (+ 78us) write_op.cc:276] Timestamp: P: 1750967859655198 usec, L: 0
0626 19:57:49.164706 (+ 486us) write_op.cc:270] Start()
0626 19:57:49.164773 (+ 67us) write_op.cc:276] Timestamp: P: 1750967859686747 usec, L: 0
0626 19:57:49.165177 (+ 404us) write_op.cc:270] Start()
0626 19:57:49.165263 (+ 86us) write_op.cc:276] Timestamp: P: 1750967859821622 usec, L: 0
0626 19:57:49.165742 (+ 479us) write_op.cc:270] Start()
0626 19:57:49.165816 (+ 74us) write_op.cc:276] Timestamp: P: 1750967859878579 usec, L: 0
0626 19:57:49.166298 (+ 482us) write_op.cc:270] Start()
0626 19:57:49.166403 (+ 105us) write_op.cc:276] Timestamp: P: 1750967859939788 usec, L: 0
0626 19:57:49.208090 (+ 41687us) write_op.cc:270] Start()
0626 19:57:49.208203 (+ 113us) write_op.cc:276] Timestamp: P: 1750967860035255 usec, L: 0
0626 19:57:49.208686 (+ 483us) write_op.cc:270] Start()
0626 19:57:49.208783 (+ 97us) write_op.cc:276] Timestamp: P: 1750967860099488 usec, L: 0
0626 19:57:49.209326 (+ 543us) write_op.cc:270] Start()
0626 19:57:49.209416 (+ 90us) write_op.cc:276] Timestamp: P: 1750967860101523 usec, L: 0
0626 19:57:49.209982 (+ 566us) write_op.cc:270] Start()
0626 19:57:49.210083 (+ 101us) write_op.cc:276] Timestamp: P: 1750967860299154 usec, L: 0
0626 19:57:49.210692 (+ 609us) write_op.cc:270] Start()
0626 19:57:49.210802 (+ 110us) write_op.cc:276] Timestamp: P: 1750967860354192 usec, L: 0
0626 19:57:49.211343 (+ 541us) write_op.cc:270] Start()
0626 19:57:49.211429 (+ 86us) write_op.cc:276] Timestamp: P: 1750967860476126 usec, L: 0
0626 19:57:49.212269 (+ 840us) write_op.cc:270] Start()
0626 19:57:49.212367 (+ 98us) write_op.cc:276] Timestamp: P: 1750967860532774 usec, L: 0
0626 19:57:49.212898 (+ 531us) write_op.cc:270] Start()
0626 19:57:49.212986 (+ 88us) write_op.cc:276] Timestamp: P: 1750967860563563 usec, L: 0
0626 19:57:49.213494 (+ 508us) write_op.cc:270] Start()
0626 19:57:49.213584 (+ 90us) write_op.cc:276] Timestamp: P: 1750967860713399 usec, L: 0
0626 19:57:49.270830 (+ 57246us) write_op.cc:270] Start()
0626 19:57:49.270931 (+ 101us) write_op.cc:276] Timestamp: P: 1750967860835332 usec, L: 0
0626 19:57:49.271542 (+ 611us) write_op.cc:270] Start()
0626 19:57:49.271620 (+ 78us) write_op.cc:276] Timestamp: P: 1750967861300314 usec, L: 0
0626 19:57:49.272040 (+ 420us) write_op.cc:270] Start()
0626 19:57:49.272105 (+ 65us) write_op.cc:276] Timestamp: P: 1750967861345169 usec, L: 0
0626 19:57:49.272540 (+ 435us) write_op.cc:270] Start()
0626 19:57:49.272616 (+ 76us) write_op.cc:276] Timestamp: P: 1750967861685323 usec, L: 0
0626 19:57:49.273015 (+ 399us) write_op.cc:270] Start()
0626 19:57:49.273084 (+ 69us) write_op.cc:276] Timestamp: P: 1750967861844641 usec, L: 0
0626 19:57:49.273504 (+ 420us) write_op.cc:270] Start()
0626 19:57:49.273571 (+ 67us) write_op.cc:276] Timestamp: P: 1750967862189048 usec, L: 0
0626 19:57:49.273974 (+ 403us) write_op.cc:270] Start()
0626 19:57:49.274040 (+ 66us) write_op.cc:276] Timestamp: P: 1750967862697322 usec, L: 0
0626 19:57:49.328810 (+ 54770us) log.cc:844] Serialized 97028 byte log entry
0626 19:57:49.329481 (+ 671us) raft_consensus.cc:1643] Marking committed up to 224
0626 19:57:49.329761 (+ 280us) log.cc:844] Serialized 14 byte log entry
0626 19:57:49.481395 (+151634us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.481419 (+ 24us) write_op.cc:454] Released schema lock
0626 19:57:49.493590 (+ 12171us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.493615 (+ 25us) write_op.cc:454] Released schema lock
0626 19:57:49.601801 (+108186us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 19:57:49.601890 (+ 89us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 19:57:49.601950 (+ 60us) raft_consensus.cc:1683] finished
0626 19:57:49.601971 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 19:57:49.602515 (+ 544us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:48.326010 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:48.326480 (+ 470us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:48.326513 (+ 33us) write_op.cc:435] Acquired schema lock
0626 19:57:48.326531 (+ 18us) tablet.cc:660] Decoding operations
0626 19:57:48.326864 (+ 333us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:48.326925 (+ 61us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:48.326954 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:48.327116 (+ 162us) tablet.cc:699] Row locks acquired
0626 19:57:48.327133 (+ 17us) write_op.cc:260] PREPARE: finished
0626 19:57:49.455341 (+1128208us) op_driver.cc:464] REPLICATION: finished
0626 19:57:49.469779 (+ 14438us) write_op.cc:301] APPLY: starting
0626 19:57:49.469844 (+ 65us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:49.469935 (+ 91us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:49.469957 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:49.470132 (+ 175us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:49.470293 (+ 161us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:49.470442 (+ 149us) write_op.cc:312] APPLY: finished
0626 19:57:49.470611 (+ 169us) log.cc:844] Serialized 22 byte log entry
0626 19:57:49.470945 (+ 334us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.471004 (+ 59us) write_op.cc:454] Released schema lock
0626 19:57:49.471034 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:49.471258 (+ 224us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.471281 (+ 23us) write_op.cc:454] Released schema lock
Related trace 'op':
0626 19:57:48.327395 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:48.327814 (+ 419us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:48.327838 (+ 24us) write_op.cc:435] Acquired schema lock
0626 19:57:48.327854 (+ 16us) tablet.cc:660] Decoding operations
0626 19:57:48.328243 (+ 389us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:48.328313 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:48.328345 (+ 32us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:48.328595 (+ 250us) tablet.cc:699] Row locks acquired
0626 19:57:48.328611 (+ 16us) write_op.cc:260] PREPARE: finished
0626 19:57:49.466989 (+1138378us) op_driver.cc:464] REPLICATION: finished
0626 19:57:49.471524 (+ 4535us) write_op.cc:301] APPLY: starting
0626 19:57:49.471573 (+ 49us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:49.471665 (+ 92us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:49.471684 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:49.471857 (+ 173us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:49.471987 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:49.472095 (+ 108us) write_op.cc:312] APPLY: finished
0626 19:57:49.472248 (+ 153us) log.cc:844] Serialized 28 byte log entry
0626 19:57:49.472544 (+ 296us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.472601 (+ 57us) write_op.cc:454] Released schema lock
0626 19:57:49.472630 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0626 19:57:48.328867 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:48.329330 (+ 463us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:48.329360 (+ 30us) write_op.cc:435] Acquired schema lock
0626 19:57:48.329379 (+ 19us) tablet.cc:660] Decoding operations
0626 19:57:48.329691 (+ 312us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:48.329752 (+ 61us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:48.329781 (+ 29us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:48.330003 (+ 222us) tablet.cc:699] Row locks acquired
0626 19:57:48.330019 (+ 16us) write_op.cc:260] PREPARE: finished
0626 19:57:49.481665 (+1151646us) op_driver.cc:464] REPLICATION: finished
0626 19:57:49.482898 (+ 1233us) write_op.cc:301] APPLY: starting
0626 19:57:49.482951 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:49.483050 (+ 99us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:49.483069 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:49.483442 (+ 373us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:49.483573 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:49.483692 (+ 119us) write_op.cc:312] APPLY: finished
0626 19:57:49.483865 (+ 173us) log.cc:844] Serialized 28 byte log entry
0626 19:57:49.484163 (+ 298us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.484225 (+ 62us) write_op.cc:454] Released schema lock
0626 19:57:49.484256 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:49.484444 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.484466 (+ 22us) write_op.cc:454] Released schema lock
Related trace 'op':
0626 19:57:48.330709 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:48.331144 (+ 435us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:48.331167 (+ 23us) write_op.cc:435] Acquired schema lock
0626 19:57:48.331182 (+ 15us) tablet.cc:660] Decoding operations
0626 19:57:48.331511 (+ 329us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:48.331570 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:48.331599 (+ 29us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:48.331836 (+ 237us) tablet.cc:699] Row locks acquired
0626 19:57:48.331849 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:57:49.481855 (+1150006us) op_driver.cc:464] REPLICATION: finished
0626 19:57:49.484667 (+ 2812us) write_op.cc:301] APPLY: starting
0626 19:57:49.484715 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:49.484811 (+ 96us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:49.484831 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:49.485200 (+ 369us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:49.485333 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:49.485441 (+ 108us) write_op.cc:312] APPLY: finished
0626 19:57:49.485599 (+ 158us) log.cc:844] Serialized 28 byte log entry
0626 19:57:49.485880 (+ 281us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.485943 (+ 63us) write_op.cc:454] Released schema lock
0626 19:57:49.485972 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:49.486660 (+ 688us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.486684 (+ 24us) write_op.cc:454] Released schema lock
Related trace 'op':
0626 19:57:48.332100 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
W20250626 19:57:49.727169 4609 leader_election.cc:343] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Not found: Tablet not found: e94a73b50b564a97bbee0f5015bb8805
I20250626 19:57:49.596520 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c5aba053ca4535ad2376b2066cfe47" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:49.603583 5366 tablet_bootstrap.cc:654] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:49.622452 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:49.696861 6121 raft_consensus.cc:491] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:49.730957 6121 raft_consensus.cc:513] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.733875 6121 leader_election.cc:290] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:49.735258 4657 tablet_service.cc:1468] Processing CreateTablet for tablet 4631acfeee24418c8fc5f8a6416cbda9 (DEFAULT_TABLE table=default.Table_91248389db7e4324a3ddc4e5874b2666 [id=ffbbbd1dc6384b4a8aa92097ecb77f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:49.735495 4911 leader_election.cc:304] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:49.737238 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4631acfeee24418c8fc5f8a6416cbda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:49.737180 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 5ba6dda26c6a443ab6a7edbe341d49d8 (DEFAULT_TABLE table=default.Table_e2b576d684c543daaa81ab824be6ba7b [id=94d4be1f4860452893c8a5c9fca8393c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:49.737661 4651 tablet_service.cc:1468] Processing CreateTablet for tablet 5ba6dda26c6a443ab6a7edbe341d49d8 (DEFAULT_TABLE table=default.Table_e2b576d684c543daaa81ab824be6ba7b [id=94d4be1f4860452893c8a5c9fca8393c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:49.738155 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba6dda26c6a443ab6a7edbe341d49d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:49.738600 4651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba6dda26c6a443ab6a7edbe341d49d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:49.742992 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27acc0eafbb04375ba1c6a59e958e6f6" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:49.744333 4755 leader_election.cc:343] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:49.746687 6048 raft_consensus.cc:2802] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:49.747239 6048 raft_consensus.cc:695] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:49.747984 6048 consensus_queue.cc:237] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.767894 5366 tablet_bootstrap.cc:492] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:49.768428 5366 ts_tablet_manager.cc:1397] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.423s user 0.014s sys 0.002s
I20250626 19:57:49.777087 5410 tablet_bootstrap.cc:492] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:49.777493 5410 ts_tablet_manager.cc:1397] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.079s user 0.010s sys 0.011s
I20250626 19:57:49.771618 4793 tablet_service.cc:1468] Processing CreateTablet for tablet 4631acfeee24418c8fc5f8a6416cbda9 (DEFAULT_TABLE table=default.Table_91248389db7e4324a3ddc4e5874b2666 [id=ffbbbd1dc6384b4a8aa92097ecb77f84]), partition=RANGE (key) PARTITION UNBOUNDED
W20250626 19:57:49.776078 4756 leader_election.cc:343] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:49.780730 4756 leader_election.cc:304] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:49.781932 6122 raft_consensus.cc:2747] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:49.784164 5366 raft_consensus.cc:357] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.786036 5410 raft_consensus.cc:357] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.786643 5410 raft_consensus.cc:383] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.786932 5410 raft_consensus.cc:738] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:49.792896 5410 consensus_queue.cc:260] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.795013 5410 ts_tablet_manager.cc:1428] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250626 19:57:49.795432 5366 raft_consensus.cc:383] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.803429 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c5aba053ca4535ad2376b2066cfe47" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:49.804210 4670 raft_consensus.cc:2391] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
W20250626 19:57:49.805076 4611 leader_election.cc:343] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Not found: Tablet not found: e94a73b50b564a97bbee0f5015bb8805
I20250626 19:57:49.805450 4611 leader_election.cc:304] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:49.808109 6046 raft_consensus.cc:2747] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:49.811754 6180 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250626 19:57:49.795886 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 5ba6dda26c6a443ab6a7edbe341d49d8 (DEFAULT_TABLE table=default.Table_e2b576d684c543daaa81ab824be6ba7b [id=94d4be1f4860452893c8a5c9fca8393c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:49.818637 5366 raft_consensus.cc:738] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:49.801898 4796 tablet_service.cc:1515] Processing DeleteTablet for tablet c990e4eca5cf4a8e8295e9599cf5be29 with delete_type TABLET_DATA_DELETED (replaced by tablet c2ff9841bbbd429b83c123e1ebdabfea) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:49.851027 5366 consensus_queue.cc:260] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.853704 5366 ts_tablet_manager.cc:1428] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.074s user 0.006s sys 0.000s
I20250626 19:57:49.856504 5795 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" instance_seqno: 1750967844408046) as {username='slave'} at 127.4.46.131:44819; Asking this server to re-register.
I20250626 19:57:49.869419 5410 tablet_bootstrap.cc:492] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:49.882752 5353 tablet_bootstrap.cc:492] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:49.883054 4793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4631acfeee24418c8fc5f8a6416cbda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:49.883203 5353 ts_tablet_manager.cc:1397] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.218s user 0.011s sys 0.014s
I20250626 19:57:49.885385 5353 raft_consensus.cc:357] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.885982 5353 raft_consensus.cc:383] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.886212 5353 raft_consensus.cc:738] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:49.887219 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba6dda26c6a443ab6a7edbe341d49d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:49.888738 5366 tablet_bootstrap.cc:492] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:49.894876 5353 consensus_queue.cc:260] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.896932 5353 ts_tablet_manager.cc:1428] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.013s user 0.002s sys 0.002s
I20250626 19:57:49.897753 5353 tablet_bootstrap.cc:492] T b852dd204f8a42cca629901723fc9247 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:49.902381 5410 tablet_bootstrap.cc:654] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:49.902110 6122 consensus_queue.cc:1035] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:49.908216 5353 tablet_bootstrap.cc:654] T b852dd204f8a42cca629901723fc9247 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:49.909972 6191 tablet_replica.cc:331] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:49.910972 6191 raft_consensus.cc:2241] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 2 LEADER]: Raft consensus shutting down.
I20250626 19:57:49.911088 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27acc0eafbb04375ba1c6a59e958e6f6" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:49.911811 6191 raft_consensus.cc:2270] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:49.912446 4973 raft_consensus.cc:3058] T 83f0431638144710b598b3f87d9a576e P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:49.915346 4756 leader_election.cc:343] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:49.915695 4756 leader_election.cc:304] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:49.915664 4970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c5aba053ca4535ad2376b2066cfe47" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:49.917487 6121 raft_consensus.cc:2747] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:49.931571 6191 ts_tablet_manager.cc:1905] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:49.932761 5366 tablet_bootstrap.cc:654] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:49.932847 4611 leader_election.cc:304] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:49.934396 6084 raft_consensus.cc:2747] T d7c5aba053ca4535ad2376b2066cfe47 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:57:49.939154 6080 raft_consensus.cc:491] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:49.939610 6080 raft_consensus.cc:513] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.941557 6080 leader_election.cc:290] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:49.944253 5353 tablet_bootstrap.cc:492] T b852dd204f8a42cca629901723fc9247 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:49.944626 5353 ts_tablet_manager.cc:1397] T b852dd204f8a42cca629901723fc9247 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.000s
I20250626 19:57:49.946713 6122 consensus_queue.cc:1035] T 83f0431638144710b598b3f87d9a576e P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:49.946739 5353 raft_consensus.cc:357] T b852dd204f8a42cca629901723fc9247 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.947350 5353 raft_consensus.cc:383] T b852dd204f8a42cca629901723fc9247 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.947585 5353 raft_consensus.cc:738] T b852dd204f8a42cca629901723fc9247 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:49.948100 5353 consensus_queue.cc:260] T b852dd204f8a42cca629901723fc9247 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:49.948819 4911 leader_election.cc:343] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Not found: Tablet not found: fc526a05990446a3b3bc973e79511476
I20250626 19:57:49.950474 5410 tablet_bootstrap.cc:492] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:49.950902 5410 ts_tablet_manager.cc:1397] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.003s
I20250626 19:57:49.951246 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc526a05990446a3b3bc973e79511476" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:49.953828 5410 raft_consensus.cc:357] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.966897 5410 raft_consensus.cc:383] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.967381 5410 raft_consensus.cc:738] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:49.968055 5410 consensus_queue.cc:260] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:49.970103 5410 ts_tablet_manager.cc:1428] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20250626 19:57:49.970947 5410 tablet_bootstrap.cc:492] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:49.974229 5353 ts_tablet_manager.cc:1428] T b852dd204f8a42cca629901723fc9247 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.029s user 0.006s sys 0.001s
W20250626 19:57:49.977314 4912 leader_election.cc:343] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:49.977905 4912 leader_election.cc:304] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:49.978622 6080 raft_consensus.cc:2747] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:49.981634 4671 raft_consensus.cc:3058] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:49.988718 5353 tablet_bootstrap.cc:492] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:50.009143 6048 raft_consensus.cc:491] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:49.994612 5410 tablet_bootstrap.cc:654] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.010923 6048 raft_consensus.cc:513] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.019008 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:50.024158 5410 tablet_bootstrap.cc:492] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:50.024561 5410 ts_tablet_manager.cc:1397] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.003s
I20250626 19:57:50.025090 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 0f834922c9de46dca48c47748913b11a (table default.Table_6251ce018aaf4ee0948f7c26518cb21f [id=594e3d715a6d42c798c44d20740d2ebb]). Expected version 1 got 0
I20250626 19:57:50.025840 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:50.026885 5410 raft_consensus.cc:357] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.027536 5410 raft_consensus.cc:383] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.027783 5410 raft_consensus.cc:738] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.028337 5410 consensus_queue.cc:260] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.031554 5366 tablet_bootstrap.cc:492] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:50.031926 5366 ts_tablet_manager.cc:1397] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.143s user 0.007s sys 0.002s
I20250626 19:57:50.034466 6195 raft_consensus.cc:491] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:50.013893 5353 tablet_bootstrap.cc:654] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.014483 5026 heartbeater.cc:461] Registering TS with master...
I20250626 19:57:50.034966 6195 raft_consensus.cc:513] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.033558 6080 consensus_queue.cc:1035] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:50.037415 6195 leader_election.cc:290] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:50.038106 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1feb6785d0c7459fbc6898b0c9d3e38c" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:50.039374 4755 leader_election.cc:343] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:50.041221 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (table default.Table_56852b6e38af4545879ff672cc528975 [id=e3400f897e9944549162854e3f92ac21]). Expected version 1 got 0
I20250626 19:57:50.042742 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:50.030668 5410 ts_tablet_manager.cc:1428] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250626 19:57:50.055212 5410 tablet_bootstrap.cc:492] T d8399c219d0e4485bb7926eefd78f210 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:50.056068 6048 leader_election.cc:290] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:50.066084 5353 tablet_bootstrap.cc:492] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:50.067528 5353 ts_tablet_manager.cc:1397] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.079s user 0.014s sys 0.003s
I20250626 19:57:50.067783 4407 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 7 candidate_status { last_received { term: 6 index: 224 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:50.070960 5366 raft_consensus.cc:357] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.071605 5366 raft_consensus.cc:383] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.072791 5353 raft_consensus.cc:357] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.073529 5353 raft_consensus.cc:383] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:49.844620 6166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "62ac80df82774b9b87c0be076b5637af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } } }
I20250626 19:57:50.073827 5353 raft_consensus.cc:738] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.074244 6166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af [sys.catalog]: This master's current role is: LEADER
I20250626 19:57:50.071859 5366 raft_consensus.cc:738] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.075795 5366 consensus_queue.cc:260] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:50.080206 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 46229.4us
I20250626 19:57:50.081703 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d43617ed2cb4a0085e59eefcf8b19f1" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:50.081826 4486 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 7 candidate_status { last_received { term: 6 index: 224 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31" is_pre_election: true
I20250626 19:57:50.082259 4816 raft_consensus.cc:2466] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:50.083213 4911 leader_election.cc:304] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:50.084656 6080 raft_consensus.cc:2802] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:50.084969 6080 raft_consensus.cc:491] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:50.085242 6080 raft_consensus.cc:3058] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.083922 5770 leader_election.cc:304] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6c58006b6be41c1bd6bdb5eb010d360; no voters: 62ac80df82774b9b87c0be076b5637af, ae949d1fcdf54e87930871ac0ae7da31
I20250626 19:57:50.086400 5410 tablet_bootstrap.cc:654] T d8399c219d0e4485bb7926eefd78f210 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.078629 5353 consensus_queue.cc:260] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.093210 5353 ts_tablet_manager.cc:1428] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20250626 19:57:50.094077 5353 tablet_bootstrap.cc:492] T 27acc0eafbb04375ba1c6a59e958e6f6 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
W20250626 19:57:50.094377 4384 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e9f09f0381eb420ca57c18a5be6f2cc2 consensus state has no leader
I20250626 19:57:50.094157 6080 raft_consensus.cc:513] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.100639 6080 leader_election.cc:290] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:50.101814 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d43617ed2cb4a0085e59eefcf8b19f1" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:50.102373 4816 raft_consensus.cc:3058] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.102573 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d43617ed2cb4a0085e59eefcf8b19f1" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:50.102855 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac]). Expected version 1 got 0
I20250626 19:57:50.103324 4670 raft_consensus.cc:2466] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:50.103600 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:50.104110 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 27acc0eafbb04375ba1c6a59e958e6f6 (table default.Table_2d1ec663bb8048e0ac15d7025383ef5d [id=30ad5642ec2d422aad5b97698297fbe9]). Expected version 1 got 0
I20250626 19:57:50.104995 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (table default.Table_8b997467fa764e8fbf273c3b1f2ccb3e [id=760c56838ee140d78aa368a3810d19a8]). Expected version 1 got 0
I20250626 19:57:50.105782 4398 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 34b2919e5f534f4295336a4384be1f64 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:50.110039 4816 raft_consensus.cc:2466] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:50.111024 4911 leader_election.cc:304] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:50.115070 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1feb6785d0c7459fbc6898b0c9d3e38c" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:50.115206 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d43617ed2cb4a0085e59eefcf8b19f1" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:50.115672 4670 raft_consensus.cc:3058] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.115610 4979 raft_consensus.cc:2466] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:50.138382 5353 tablet_bootstrap.cc:654] T 27acc0eafbb04375ba1c6a59e958e6f6 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.139333 6151 raft_consensus.cc:2802] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:50.139803 6151 raft_consensus.cc:695] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:50.140552 6151 consensus_queue.cc:237] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.156584 5410 tablet_bootstrap.cc:492] T d8399c219d0e4485bb7926eefd78f210 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:50.157258 5410 ts_tablet_manager.cc:1397] T d8399c219d0e4485bb7926eefd78f210 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.112s user 0.010s sys 0.003s
I20250626 19:57:50.160629 5410 raft_consensus.cc:357] T d8399c219d0e4485bb7926eefd78f210 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:50.161446 5410 raft_consensus.cc:383] T d8399c219d0e4485bb7926eefd78f210 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.161743 5410 raft_consensus.cc:738] T d8399c219d0e4485bb7926eefd78f210 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.162833 5410 consensus_queue.cc:260] T d8399c219d0e4485bb7926eefd78f210 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:50.164842 5410 ts_tablet_manager.cc:1428] T d8399c219d0e4485bb7926eefd78f210 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250626 19:57:50.165894 5410 tablet_bootstrap.cc:492] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:50.179826 5410 tablet_bootstrap.cc:654] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.188995 5366 ts_tablet_manager.cc:1428] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.157s user 0.006s sys 0.000s
I20250626 19:57:50.189980 5366 tablet_bootstrap.cc:492] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:50.190012 6191 ts_tablet_manager.cc:1918] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250626 19:57:50.191426 4756 leader_election.cc:304] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:50.191741 6191 log.cc:1199] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/c990e4eca5cf4a8e8295e9599cf5be29
I20250626 19:57:50.192720 6191 ts_tablet_manager.cc:1939] T c990e4eca5cf4a8e8295e9599cf5be29 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:50.193341 6121 raft_consensus.cc:2802] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:50.193683 6121 raft_consensus.cc:491] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:50.193976 6121 raft_consensus.cc:3058] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.200994 4670 raft_consensus.cc:2466] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:50.213631 5366 tablet_bootstrap.cc:654] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.215664 6121 raft_consensus.cc:513] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.219758 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1feb6785d0c7459fbc6898b0c9d3e38c" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
W20250626 19:57:50.220992 4755 leader_election.cc:343] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:50.227223 5924 consensus_queue.cc:1035] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:50.230633 5410 tablet_bootstrap.cc:492] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:50.231057 5410 ts_tablet_manager.cc:1397] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.065s user 0.010s sys 0.008s
I20250626 19:57:50.233366 5410 raft_consensus.cc:357] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.234007 5410 raft_consensus.cc:383] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.234262 5410 raft_consensus.cc:738] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.235014 5410 consensus_queue.cc:260] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.239656 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1feb6785d0c7459fbc6898b0c9d3e38c" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:50.240198 4979 raft_consensus.cc:3058] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.243468 6121 leader_election.cc:290] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:50.252786 4979 raft_consensus.cc:2466] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:50.253080 6141 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 19:57:50.256307 5410 ts_tablet_manager.cc:1428] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.025s user 0.005s sys 0.001s
I20250626 19:57:50.257906 4756 leader_election.cc:304] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:50.258456 5410 tablet_bootstrap.cc:492] T cf5c4cd6c2f64877a9e348dedb9820a3 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:50.258994 6121 raft_consensus.cc:2802] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:50.259608 6121 raft_consensus.cc:695] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:50.260149 5796 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4d3be519a0044a2f921a503412f72e86" instance_seqno: 1750967840032510) as {username='slave'} at 127.4.46.129:48913; Asking this server to re-register.
I20250626 19:57:50.260362 6121 consensus_queue.cc:237] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.263810 4727 heartbeater.cc:461] Registering TS with master...
I20250626 19:57:50.264465 5793 ts_manager.cc:194] Registered new tserver with Master: 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:50.267654 5797 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" instance_seqno: 1750967842396780) as {username='slave'} at 127.4.46.130:38237; Asking this server to re-register.
I20250626 19:57:50.270032 5410 tablet_bootstrap.cc:654] T cf5c4cd6c2f64877a9e348dedb9820a3 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.273932 4870 heartbeater.cc:461] Registering TS with master...
I20250626 19:57:50.282234 5795 ts_manager.cc:194] Registered new tserver with Master: 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:50.282682 5366 tablet_bootstrap.cc:492] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:50.283123 5366 ts_tablet_manager.cc:1397] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.093s user 0.008s sys 0.003s
I20250626 19:57:50.285382 5366 raft_consensus.cc:357] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.285989 5366 raft_consensus.cc:383] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.286255 5366 raft_consensus.cc:738] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.286742 5410 tablet_bootstrap.cc:492] T cf5c4cd6c2f64877a9e348dedb9820a3 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:50.287178 5410 ts_tablet_manager.cc:1397] T cf5c4cd6c2f64877a9e348dedb9820a3 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.010s
I20250626 19:57:50.287006 5366 consensus_queue.cc:260] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.289427 5410 raft_consensus.cc:357] T cf5c4cd6c2f64877a9e348dedb9820a3 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.293889 5410 raft_consensus.cc:383] T cf5c4cd6c2f64877a9e348dedb9820a3 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.294142 5410 raft_consensus.cc:738] T cf5c4cd6c2f64877a9e348dedb9820a3 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.295794 5410 consensus_queue.cc:260] T cf5c4cd6c2f64877a9e348dedb9820a3 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.297829 5410 ts_tablet_manager.cc:1428] T cf5c4cd6c2f64877a9e348dedb9820a3 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.010s user 0.003s sys 0.001s
I20250626 19:57:50.297981 5366 ts_tablet_manager.cc:1428] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250626 19:57:50.298657 5410 tablet_bootstrap.cc:492] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:50.299243 5366 tablet_bootstrap.cc:492] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:50.301542 5797 ts_manager.cc:194] Registered new tserver with Master: 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:50.303893 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]) successfully deleted
I20250626 19:57:50.304133 5366 tablet_bootstrap.cc:654] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.312024 5410 tablet_bootstrap.cc:654] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.314939 6121 raft_consensus.cc:491] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:50.317437 6121 raft_consensus.cc:513] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.319893 6121 leader_election.cc:290] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:50.320755 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f70c620bf4474e59aa1f86761aca0daa" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:50.321971 4755 leader_election.cc:343] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:50.322882 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f70c620bf4474e59aa1f86761aca0daa" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:50.324431 4756 leader_election.cc:343] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:50.324823 4756 leader_election.cc:304] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:50.325378 5366 tablet_bootstrap.cc:492] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:50.325731 5366 ts_tablet_manager.cc:1397] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.007s
I20250626 19:57:50.327929 5366 raft_consensus.cc:357] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:50.328516 5366 raft_consensus.cc:383] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.328775 5366 raft_consensus.cc:738] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.330078 6121 raft_consensus.cc:2747] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:50.340643 5410 tablet_bootstrap.cc:492] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:50.341223 5410 ts_tablet_manager.cc:1397] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.000s
I20250626 19:57:50.344578 5410 raft_consensus.cc:357] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.345332 5410 raft_consensus.cc:383] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.345621 5410 raft_consensus.cc:738] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.346279 5410 consensus_queue.cc:260] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.349073 5353 tablet_bootstrap.cc:492] T 27acc0eafbb04375ba1c6a59e958e6f6 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:50.349437 5353 ts_tablet_manager.cc:1397] T 27acc0eafbb04375ba1c6a59e958e6f6 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.256s user 0.030s sys 0.050s
I20250626 19:57:50.350469 5410 ts_tablet_manager.cc:1428] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20250626 19:57:50.351668 5353 raft_consensus.cc:357] T 27acc0eafbb04375ba1c6a59e958e6f6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.352260 5353 raft_consensus.cc:383] T 27acc0eafbb04375ba1c6a59e958e6f6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.352305 5410 tablet_bootstrap.cc:492] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:50.352499 5353 raft_consensus.cc:738] T 27acc0eafbb04375ba1c6a59e958e6f6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.329381 5366 consensus_queue.cc:260] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:50.353034 5353 consensus_queue.cc:260] T 27acc0eafbb04375ba1c6a59e958e6f6 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.354990 5353 ts_tablet_manager.cc:1428] T 27acc0eafbb04375ba1c6a59e958e6f6 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250626 19:57:50.358985 5410 tablet_bootstrap.cc:654] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.364023 5353 tablet_bootstrap.cc:492] T 075dbe46bab14d40a8008ce5c24bcf38 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:50.364959 5366 ts_tablet_manager.cc:1428] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.039s user 0.005s sys 0.001s
I20250626 19:57:50.371850 5410 tablet_bootstrap.cc:492] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:50.372342 5410 ts_tablet_manager.cc:1397] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.003s
I20250626 19:57:50.374799 5410 raft_consensus.cc:357] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.375653 5366 tablet_bootstrap.cc:492] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:50.375661 5410 raft_consensus.cc:383] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.376176 5410 raft_consensus.cc:738] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.376833 5410 consensus_queue.cc:260] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.378913 5410 ts_tablet_manager.cc:1428] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250626 19:57:50.380113 5410 tablet_bootstrap.cc:492] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:50.380846 6046 raft_consensus.cc:491] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:50.381289 6046 raft_consensus.cc:513] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.383086 6046 leader_election.cc:290] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:50.384402 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc526a05990446a3b3bc973e79511476" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:50.385011 4973 raft_consensus.cc:2466] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
W20250626 19:57:50.385737 4609 leader_election.cc:343] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Not found: Tablet not found: fc526a05990446a3b3bc973e79511476
I20250626 19:57:50.386197 4611 leader_election.cc:304] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters: 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:50.391327 6085 raft_consensus.cc:2802] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:50.391667 6085 raft_consensus.cc:491] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:50.391959 6085 raft_consensus.cc:3058] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.394284 5410 tablet_bootstrap.cc:654] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.405308 5366 tablet_bootstrap.cc:654] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.420436 6084 raft_consensus.cc:491] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:50.420915 6084 raft_consensus.cc:513] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.421608 5353 tablet_bootstrap.cc:654] T 075dbe46bab14d40a8008ce5c24bcf38 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.423498 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27acc0eafbb04375ba1c6a59e958e6f6" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:50.423841 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27acc0eafbb04375ba1c6a59e958e6f6" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:50.424043 4979 raft_consensus.cc:2466] T 27acc0eafbb04375ba1c6a59e958e6f6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:50.424433 4816 raft_consensus.cc:2466] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:50.422688 6084 leader_election.cc:290] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:50.425021 4611 leader_election.cc:304] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:50.425678 6084 raft_consensus.cc:2802] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:50.410626 6085 raft_consensus.cc:513] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.425992 6084 raft_consensus.cc:491] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:50.426409 6084 raft_consensus.cc:3058] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.433284 6084 raft_consensus.cc:513] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.442498 5366 tablet_bootstrap.cc:492] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:50.442970 5366 ts_tablet_manager.cc:1397] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.004s
I20250626 19:57:50.445423 5366 raft_consensus.cc:357] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.446046 5366 raft_consensus.cc:383] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.450925 5366 raft_consensus.cc:738] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.451790 5366 consensus_queue.cc:260] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.459400 5366 ts_tablet_manager.cc:1428] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.016s user 0.004s sys 0.001s
I20250626 19:57:50.460273 5366 tablet_bootstrap.cc:492] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:50.510351 6048 raft_consensus.cc:491] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:50.510869 6048 raft_consensus.cc:513] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:50.512687 6048 leader_election.cc:290] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:50.514912 5366 tablet_bootstrap.cc:654] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.514712 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8399c219d0e4485bb7926eefd78f210" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:50.515311 4816 raft_consensus.cc:2466] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:50.516561 4911 leader_election.cc:304] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:50.517354 6048 raft_consensus.cc:2802] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:50.517675 6048 raft_consensus.cc:491] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:50.518039 6048 raft_consensus.cc:3058] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.551826 6048 raft_consensus.cc:513] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:50.568621 6121 raft_consensus.cc:491] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:50.569121 6121 raft_consensus.cc:513] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.571065 6121 leader_election.cc:290] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:50.573643 6046 raft_consensus.cc:491] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:50.574095 6046 raft_consensus.cc:513] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.575517 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a6058961bc7450ab489b26158e7dd17" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:50.576493 4756 leader_election.cc:343] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:50.579420 6046 leader_election.cc:290] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:50.577267 6085 leader_election.cc:290] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:50.590929 6084 leader_election.cc:290] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:50.591881 5410 tablet_bootstrap.cc:492] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:50.592269 5410 ts_tablet_manager.cc:1397] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.213s user 0.025s sys 0.044s
I20250626 19:57:50.611595 4397 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:50.613154 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:50.615052 5353 tablet_bootstrap.cc:492] T 075dbe46bab14d40a8008ce5c24bcf38 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:50.615455 5353 ts_tablet_manager.cc:1397] T 075dbe46bab14d40a8008ce5c24bcf38 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.252s user 0.003s sys 0.006s
I20250626 19:57:50.617784 5353 raft_consensus.cc:357] T 075dbe46bab14d40a8008ce5c24bcf38 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:50.618377 5353 raft_consensus.cc:383] T 075dbe46bab14d40a8008ce5c24bcf38 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.594413 5410 raft_consensus.cc:357] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:50.619405 5410 raft_consensus.cc:383] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.619696 5410 raft_consensus.cc:738] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.620265 5410 consensus_queue.cc:260] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:50.622556 5410 ts_tablet_manager.cc:1428] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.030s user 0.003s sys 0.002s
I20250626 19:57:50.623446 5410 tablet_bootstrap.cc:492] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:50.626490 6231 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 62ac80df82774b9b87c0be076b5637af)
I20250626 19:57:50.627081 6231 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:50.629135 6231 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:50.630677 4407 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 7 candidate_status { last_received { term: 6 index: 289 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:50.632647 5836 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 7 candidate_status { last_received { term: 6 index: 289 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:50.634131 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc526a05990446a3b3bc973e79511476" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:50.635187 4971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b852dd204f8a42cca629901723fc9247" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:50.636092 6084 raft_consensus.cc:491] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:50.635973 4971 raft_consensus.cc:2466] T b852dd204f8a42cca629901723fc9247 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:50.636533 6084 raft_consensus.cc:513] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.637507 4975 raft_consensus.cc:3058] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.638662 6084 leader_election.cc:290] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:50.640841 6085 raft_consensus.cc:491] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:50.641348 6085 raft_consensus.cc:513] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.641633 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27acc0eafbb04375ba1c6a59e958e6f6" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:50.641633 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27acc0eafbb04375ba1c6a59e958e6f6" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:50.642158 4819 raft_consensus.cc:3058] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.642158 4979 raft_consensus.cc:3058] T 27acc0eafbb04375ba1c6a59e958e6f6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.643376 6085 leader_election.cc:290] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:50.643790 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b852dd204f8a42cca629901723fc9247" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:50.644305 4821 raft_consensus.cc:2466] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:50.645190 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a6058961bc7450ab489b26158e7dd17" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:50.649050 4755 leader_election.cc:343] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:50.649008 4819 raft_consensus.cc:2466] T 27acc0eafbb04375ba1c6a59e958e6f6 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:50.649523 4755 leader_election.cc:304] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:50.650496 6121 raft_consensus.cc:2747] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:50.650921 4397 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 0f834922c9de46dca48c47748913b11a (table default.Table_6251ce018aaf4ee0948f7c26518cb21f [id=594e3d715a6d42c798c44d20740d2ebb]). Expected version 1 got 0
I20250626 19:57:50.659806 4397 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (table default.Table_56852b6e38af4545879ff672cc528975 [id=e3400f897e9944549162854e3f92ac21]). Expected version 1 got 0
I20250626 19:57:50.661092 5366 tablet_bootstrap.cc:492] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:50.661504 5366 ts_tablet_manager.cc:1397] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.201s user 0.016s sys 0.044s
I20250626 19:57:50.662268 4970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f807cf54bbb146ff989762c18410e6bb" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:50.662935 4970 raft_consensus.cc:2466] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:50.663111 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8399c219d0e4485bb7926eefd78f210" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:50.663650 4819 raft_consensus.cc:3058] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.662930 6048 leader_election.cc:290] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:50.665153 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8399c219d0e4485bb7926eefd78f210" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:50.665681 4670 raft_consensus.cc:2466] T d8399c219d0e4485bb7926eefd78f210 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:50.667098 4611 leader_election.cc:304] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:50.668045 4611 leader_election.cc:304] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:50.664026 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4aee96ad3454d65b974c867605c4cdc" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:50.669063 4973 raft_consensus.cc:2391] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:50.669067 6229 raft_consensus.cc:2802] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:50.669380 6229 raft_consensus.cc:491] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:50.670157 6229 raft_consensus.cc:3058] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.670246 5366 raft_consensus.cc:357] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.672632 6228 raft_consensus.cc:2802] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:50.672967 6228 raft_consensus.cc:491] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:50.673024 5366 raft_consensus.cc:383] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.673265 6228 raft_consensus.cc:3058] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.673329 5366 raft_consensus.cc:738] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.674268 4397 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:50.674013 5366 consensus_queue.cc:260] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.675521 4979 raft_consensus.cc:2466] T 27acc0eafbb04375ba1c6a59e958e6f6 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:50.676010 4975 raft_consensus.cc:2466] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:50.676440 5366 ts_tablet_manager.cc:1428] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250626 19:57:50.677340 4819 raft_consensus.cc:2466] T d8399c219d0e4485bb7926eefd78f210 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:50.677412 4611 leader_election.cc:304] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:50.678619 5366 tablet_bootstrap.cc:492] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:50.678871 4911 leader_election.cc:304] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:50.679903 4611 leader_election.cc:304] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:50.681159 6085 raft_consensus.cc:2802] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:50.681732 6046 raft_consensus.cc:2802] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:50.682169 6046 raft_consensus.cc:695] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:50.683005 6046 consensus_queue.cc:237] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.684396 5366 tablet_bootstrap.cc:654] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.685935 6085 raft_consensus.cc:695] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:50.686791 6085 consensus_queue.cc:237] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.694641 5366 tablet_bootstrap.cc:492] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:50.695045 5366 ts_tablet_manager.cc:1397] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.006s
I20250626 19:57:50.618718 5353 raft_consensus.cc:738] T 075dbe46bab14d40a8008ce5c24bcf38 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.697167 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8399c219d0e4485bb7926eefd78f210" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:50.697058 5366 raft_consensus.cc:357] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.697870 4670 raft_consensus.cc:3058] T d8399c219d0e4485bb7926eefd78f210 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.697634 5366 raft_consensus.cc:383] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.698393 5366 raft_consensus.cc:738] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.703359 4397 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:50.703969 4670 raft_consensus.cc:2466] T d8399c219d0e4485bb7926eefd78f210 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:50.695868 6048 raft_consensus.cc:2802] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:50.705456 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4aee96ad3454d65b974c867605c4cdc" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:50.706144 4819 raft_consensus.cc:2391] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 93f885c6802341c889a40e9c20fbfff1 in this term.
I20250626 19:57:50.705907 5353 consensus_queue.cc:260] T 075dbe46bab14d40a8008ce5c24bcf38 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:50.707093 4397 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 27acc0eafbb04375ba1c6a59e958e6f6 (table default.Table_2d1ec663bb8048e0ac15d7025383ef5d [id=30ad5642ec2d422aad5b97698297fbe9]). Expected version 1 got 0
I20250626 19:57:50.709740 5366 consensus_queue.cc:260] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.711862 5366 ts_tablet_manager.cc:1428] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20250626 19:57:50.714351 5366 tablet_bootstrap.cc:492] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:50.715874 4397 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:50.645699 4464 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31; no voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:50.717866 6048 raft_consensus.cc:695] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
W20250626 19:57:50.717895 4464 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.189:45039, user_credentials={real_user=slave}} blocked reactor thread for 72535.5us
I20250626 19:57:50.722488 5366 tablet_bootstrap.cc:654] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.722548 6048 consensus_queue.cc:237] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:50.730119 5353 ts_tablet_manager.cc:1428] T 075dbe46bab14d40a8008ce5c24bcf38 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.114s user 0.004s sys 0.003s
W20250626 19:57:50.734123 4609 leader_election.cc:343] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Not found: Tablet not found: fc526a05990446a3b3bc973e79511476
I20250626 19:57:50.754593 4397 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac]). Expected version 1 got 0
I20250626 19:57:50.784083 4397 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (table default.Table_8b997467fa764e8fbf273c3b1f2ccb3e [id=760c56838ee140d78aa368a3810d19a8]). Expected version 1 got 0
W20250626 19:57:50.789065 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 61562us
I20250626 19:57:50.789938 5353 tablet_bootstrap.cc:492] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:50.796634 4397 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 34b2919e5f534f4295336a4384be1f64 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:50.798538 4609 leader_election.cc:304] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:50.803910 6231 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 19:57:50.806154 6046 raft_consensus.cc:3058] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.806468 5410 tablet_bootstrap.cc:654] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.803256 6228 raft_consensus.cc:513] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.832245 6228 leader_election.cc:290] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:50.820143 6229 raft_consensus.cc:513] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.836455 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f807cf54bbb146ff989762c18410e6bb" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:50.837040 4979 raft_consensus.cc:3058] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:50.837980 4609 leader_election.cc:343] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Not found: Tablet not found: f807cf54bbb146ff989762c18410e6bb
I20250626 19:57:50.841194 6180 catalog_manager.cc:1261] Loaded cluster ID: 845ffa7fa66f40c8abadc28aed775425
I20250626 19:57:50.841457 6180 catalog_manager.cc:1554] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: loading cluster ID for follower catalog manager: success
I20250626 19:57:50.842013 6046 raft_consensus.cc:2747] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:50.850385 4979 raft_consensus.cc:2466] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:50.854274 6229 leader_election.cc:290] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:50.868469 5366 tablet_bootstrap.cc:492] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:50.868885 5366 ts_tablet_manager.cc:1397] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.155s user 0.038s sys 0.035s
I20250626 19:57:50.871785 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b852dd204f8a42cca629901723fc9247" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:50.872326 4815 raft_consensus.cc:3058] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.872623 5366 raft_consensus.cc:357] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.873229 5366 raft_consensus.cc:383] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.873459 5366 raft_consensus.cc:738] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.878916 5366 consensus_queue.cc:260] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.880179 4815 raft_consensus.cc:2466] T b852dd204f8a42cca629901723fc9247 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:50.880592 4611 leader_election.cc:304] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:50.887780 6228 raft_consensus.cc:2802] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:50.888208 6228 raft_consensus.cc:695] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:50.888923 6228 consensus_queue.cc:237] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.892128 5353 tablet_bootstrap.cc:654] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.901335 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b852dd204f8a42cca629901723fc9247" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:50.901957 5366 ts_tablet_manager.cc:1428] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20250626 19:57:50.901965 4979 raft_consensus.cc:3058] T b852dd204f8a42cca629901723fc9247 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.902822 5366 tablet_bootstrap.cc:492] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:50.913312 6245 raft_consensus.cc:491] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 62ac80df82774b9b87c0be076b5637af)
I20250626 19:57:50.913894 6245 raft_consensus.cc:513] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:50.918885 6046 raft_consensus.cc:491] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:50.923750 5366 tablet_bootstrap.cc:654] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.927774 4979 raft_consensus.cc:2466] T b852dd204f8a42cca629901723fc9247 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:50.931824 4611 leader_election.cc:304] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:50.932644 6228 raft_consensus.cc:2802] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:50.933123 6228 raft_consensus.cc:695] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:50.934327 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:50.933965 6228 consensus_queue.cc:237] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.934582 4407 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 7 candidate_status { last_received { term: 6 index: 289 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:50.935813 4487 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" candidate_term: 7 candidate_status { last_received { term: 6 index: 289 } } ignore_live_leader: false dest_uuid: "ae949d1fcdf54e87930871ac0ae7da31" is_pre_election: true
I20250626 19:57:50.940479 5410 tablet_bootstrap.cc:492] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:50.940865 5410 ts_tablet_manager.cc:1397] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.318s user 0.009s sys 0.007s
I20250626 19:57:50.945653 5410 raft_consensus.cc:357] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.946293 5410 raft_consensus.cc:383] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.946170 5770 leader_election.cc:304] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6c58006b6be41c1bd6bdb5eb010d360; no voters: 62ac80df82774b9b87c0be076b5637af, ae949d1fcdf54e87930871ac0ae7da31
I20250626 19:57:50.946543 5410 raft_consensus.cc:738] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.947268 6245 leader_election.cc:290] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039), ae949d1fcdf54e87930871ac0ae7da31 (127.4.46.188:40801)
I20250626 19:57:50.947960 6245 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 19:57:50.957790 5353 tablet_bootstrap.cc:492] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:50.958240 5353 ts_tablet_manager.cc:1397] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.182s user 0.010s sys 0.000s
I20250626 19:57:50.919564 6046 raft_consensus.cc:513] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:50.962244 6046 leader_election.cc:290] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:50.963289 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "075dbe46bab14d40a8008ce5c24bcf38" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:50.963876 4979 raft_consensus.cc:2466] T 075dbe46bab14d40a8008ce5c24bcf38 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:50.964061 5353 raft_consensus.cc:357] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.964656 5353 raft_consensus.cc:383] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:50.964895 5353 raft_consensus.cc:738] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:50.964939 4611 leader_election.cc:304] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:50.965487 5353 consensus_queue.cc:260] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:50.966310 4609 leader_election.cc:343] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Not found: Tablet not found: f807cf54bbb146ff989762c18410e6bb
I20250626 19:57:50.967629 6229 raft_consensus.cc:2802] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:50.967945 6229 raft_consensus.cc:491] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:50.968240 6229 raft_consensus.cc:3058] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:50.972441 6121 raft_consensus.cc:491] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:50.972893 6121 raft_consensus.cc:513] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:50.976330 5353 ts_tablet_manager.cc:1428] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.018s user 0.000s sys 0.004s
I20250626 19:57:50.976346 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ea7179912284fef8d2165cdfaf4a21a" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:50.976821 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ea7179912284fef8d2165cdfaf4a21a" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:50.977197 5353 tablet_bootstrap.cc:492] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
W20250626 19:57:50.978289 4756 leader_election.cc:343] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:50.979565 5410 consensus_queue.cc:260] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:50.980350 4755 leader_election.cc:343] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:50.980713 4755 leader_election.cc:304] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:50.981714 5410 ts_tablet_manager.cc:1428] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20250626 19:57:50.974650 6121 leader_election.cc:290] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:50.982314 6121 raft_consensus.cc:2747] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:50.982545 5410 tablet_bootstrap.cc:492] T de0724ba71824a5daaf6f3708b4431b7 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:50.983278 5366 tablet_bootstrap.cc:492] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:50.984133 5366 ts_tablet_manager.cc:1397] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.001s
I20250626 19:57:50.987363 5353 tablet_bootstrap.cc:654] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:50.993745 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:50.994786 6180 catalog_manager.cc:1576] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: acquiring CA information for follower catalog manager: success
I20250626 19:57:51.005069 5353 tablet_bootstrap.cc:492] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:51.005492 5353 ts_tablet_manager.cc:1397] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.000s
I20250626 19:57:50.990170 6229 raft_consensus.cc:513] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.007112 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:50.986452 5366 raft_consensus.cc:357] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.007954 5366 raft_consensus.cc:383] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.007839 5353 raft_consensus.cc:357] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.008282 5366 raft_consensus.cc:738] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:51.008430 5353 raft_consensus.cc:383] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.008301 6229 leader_election.cc:290] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:51.008708 5353 raft_consensus.cc:738] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
W20250626 19:57:51.009147 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 49393.4us
I20250626 19:57:51.009295 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "075dbe46bab14d40a8008ce5c24bcf38" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:51.009938 4979 raft_consensus.cc:3058] T 075dbe46bab14d40a8008ce5c24bcf38 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:51.009373 5366 consensus_queue.cc:260] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.012193 6241 consensus_queue.cc:1035] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250626 19:57:51.016093 6241 raft_consensus.cc:491] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:51.016515 6241 raft_consensus.cc:513] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.021575 4979 raft_consensus.cc:2466] T 075dbe46bab14d40a8008ce5c24bcf38 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:51.022810 4611 leader_election.cc:304] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:51.009220 5353 consensus_queue.cc:260] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.028306 5366 ts_tablet_manager.cc:1428] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.044s user 0.003s sys 0.003s
I20250626 19:57:51.029326 5353 ts_tablet_manager.cc:1428] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.024s user 0.006s sys 0.001s
I20250626 19:57:51.030174 5353 tablet_bootstrap.cc:492] T de0724ba71824a5daaf6f3708b4431b7 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:51.043306 5353 tablet_bootstrap.cc:654] T de0724ba71824a5daaf6f3708b4431b7 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:51.046554 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 0f834922c9de46dca48c47748913b11a (table default.Table_6251ce018aaf4ee0948f7c26518cb21f [id=594e3d715a6d42c798c44d20740d2ebb]). Expected version 1 got 0
I20250626 19:57:51.048256 6084 raft_consensus.cc:2802] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:51.048741 6084 raft_consensus.cc:695] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:51.049463 6084 consensus_queue.cc:237] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
W20250626 19:57:51.050338 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 82945.3us
I20250626 19:57:51.056003 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "075dbe46bab14d40a8008ce5c24bcf38" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:51.056564 4821 raft_consensus.cc:2466] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:51.057492 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "075dbe46bab14d40a8008ce5c24bcf38" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:51.057901 4821 raft_consensus.cc:3058] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:51.064551 4821 raft_consensus.cc:2466] T 075dbe46bab14d40a8008ce5c24bcf38 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:51.070803 6241 leader_election.cc:290] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:51.075764 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (table default.Table_56852b6e38af4545879ff672cc528975 [id=e3400f897e9944549162854e3f92ac21]). Expected version 1 got 0
I20250626 19:57:51.076905 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:51.078451 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:51.084631 6121 consensus_queue.cc:1035] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:51.087466 6228 raft_consensus.cc:491] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:51.087983 6228 raft_consensus.cc:513] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.089064 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5c4cd6c2f64877a9e348dedb9820a3" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:51.089648 4821 raft_consensus.cc:2466] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:51.091255 6228 leader_election.cc:290] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:51.092293 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1feb6785d0c7459fbc6898b0c9d3e38c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:51.092564 4911 leader_election.cc:304] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:51.096400 5353 tablet_bootstrap.cc:492] T de0724ba71824a5daaf6f3708b4431b7 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:51.097015 5353 ts_tablet_manager.cc:1397] T de0724ba71824a5daaf6f3708b4431b7 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.000s
I20250626 19:57:51.098630 6048 raft_consensus.cc:2802] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:51.099001 6048 raft_consensus.cc:491] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:51.099324 6048 raft_consensus.cc:3058] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:51.100790 6180 catalog_manager.cc:1604] T 00000000000000000000000000000000 P e6c58006b6be41c1bd6bdb5eb010d360: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 19:57:51.104511 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5c4cd6c2f64877a9e348dedb9820a3" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:51.105046 4670 raft_consensus.cc:2466] T cf5c4cd6c2f64877a9e348dedb9820a3 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:51.106251 6048 raft_consensus.cc:513] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.109684 6121 raft_consensus.cc:491] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:51.110095 6121 raft_consensus.cc:513] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.112557 6121 leader_election.cc:290] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:51.112870 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de0724ba71824a5daaf6f3708b4431b7" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:51.109090 5353 raft_consensus.cc:357] T de0724ba71824a5daaf6f3708b4431b7 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.113933 5353 raft_consensus.cc:383] T de0724ba71824a5daaf6f3708b4431b7 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.113837 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de0724ba71824a5daaf6f3708b4431b7" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:51.114164 5353 raft_consensus.cc:738] T de0724ba71824a5daaf6f3708b4431b7 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
W20250626 19:57:51.115303 4755 leader_election.cc:343] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:51.115830 5410 tablet_bootstrap.cc:654] T de0724ba71824a5daaf6f3708b4431b7 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:51.115335 4399 catalog_manager.cc:5582] T 125f7265258e48368381aa54559eb510 P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:51.116040 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5c4cd6c2f64877a9e348dedb9820a3" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:51.116537 4821 raft_consensus.cc:3058] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:51.117473 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 27acc0eafbb04375ba1c6a59e958e6f6 (table default.Table_2d1ec663bb8048e0ac15d7025383ef5d [id=30ad5642ec2d422aad5b97698297fbe9]). Expected version 1 got 0
I20250626 19:57:51.120780 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5c4cd6c2f64877a9e348dedb9820a3" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:51.115101 6048 leader_election.cc:290] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:51.121266 4670 raft_consensus.cc:3058] T cf5c4cd6c2f64877a9e348dedb9820a3 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:51.122215 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:51.121730 5353 consensus_queue.cc:260] T de0724ba71824a5daaf6f3708b4431b7 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.122946 4975 raft_consensus.cc:2466] T de0724ba71824a5daaf6f3708b4431b7 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:51.124924 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac]). Expected version 1 got 0
I20250626 19:57:51.124887 4756 leader_election.cc:304] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:51.126818 6242 consensus_queue.cc:1035] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:51.127146 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1feb6785d0c7459fbc6898b0c9d3e38c" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:51.128871 6241 consensus_queue.cc:1035] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:51.129525 6121 raft_consensus.cc:2802] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:51.129895 6121 raft_consensus.cc:491] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:51.130242 6121 raft_consensus.cc:3058] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:51.143669 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 34b2919e5f534f4295336a4384be1f64 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:51.144610 4670 raft_consensus.cc:2466] T cf5c4cd6c2f64877a9e348dedb9820a3 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:51.145736 5366 tablet_bootstrap.cc:492] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:51.151584 5366 tablet_bootstrap.cc:654] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:51.151377 4609 leader_election.cc:304] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:51.152076 6084 raft_consensus.cc:2747] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:51.160576 4399 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (table default.Table_8b997467fa764e8fbf273c3b1f2ccb3e [id=760c56838ee140d78aa368a3810d19a8]). Expected version 1 got 0
I20250626 19:57:51.165943 4912 leader_election.cc:304] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:51.166929 6048 raft_consensus.cc:2802] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:51.168385 6048 raft_consensus.cc:695] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:51.171403 5353 ts_tablet_manager.cc:1428] T de0724ba71824a5daaf6f3708b4431b7 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.074s user 0.006s sys 0.000s
I20250626 19:57:51.172335 5353 tablet_bootstrap.cc:492] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:51.169454 6048 consensus_queue.cc:237] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.187294 5410 tablet_bootstrap.cc:492] T de0724ba71824a5daaf6f3708b4431b7 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:51.187744 5410 ts_tablet_manager.cc:1397] T de0724ba71824a5daaf6f3708b4431b7 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.205s user 0.009s sys 0.000s
I20250626 19:57:51.188001 4669 raft_consensus.cc:3058] T 1feb6785d0c7459fbc6898b0c9d3e38c P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:51.190073 5410 raft_consensus.cc:357] T de0724ba71824a5daaf6f3708b4431b7 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.196915 6251 consensus_queue.cc:1035] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:51.202510 5366 tablet_bootstrap.cc:492] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:51.203024 5366 ts_tablet_manager.cc:1397] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.001s
I20250626 19:57:51.213182 4821 raft_consensus.cc:2466] T cf5c4cd6c2f64877a9e348dedb9820a3 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:51.224161 6121 raft_consensus.cc:513] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.190707 5410 raft_consensus.cc:383] T de0724ba71824a5daaf6f3708b4431b7 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.227221 5410 raft_consensus.cc:738] T de0724ba71824a5daaf6f3708b4431b7 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:51.228513 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de0724ba71824a5daaf6f3708b4431b7" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:51.229614 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de0724ba71824a5daaf6f3708b4431b7" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:51.230301 4979 raft_consensus.cc:3058] T de0724ba71824a5daaf6f3708b4431b7 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:51.234247 6121 leader_election.cc:290] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:51.239499 4979 raft_consensus.cc:2466] T de0724ba71824a5daaf6f3708b4431b7 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:51.242630 4756 leader_election.cc:304] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:51.243708 6121 raft_consensus.cc:2802] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:51.244146 6121 raft_consensus.cc:695] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:51.244915 6121 consensus_queue.cc:237] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.254161 5366 raft_consensus.cc:357] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.255147 5366 raft_consensus.cc:383] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.255553 5366 raft_consensus.cc:738] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:51.256454 5366 consensus_queue.cc:260] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.259200 5366 ts_tablet_manager.cc:1428] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.056s user 0.004s sys 0.002s
I20250626 19:57:51.267853 6048 consensus_queue.cc:1035] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:51.228648 5410 consensus_queue.cc:260] T de0724ba71824a5daaf6f3708b4431b7 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.276346 4671 raft_consensus.cc:3058] T de0724ba71824a5daaf6f3708b4431b7 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:51.278321 5410 ts_tablet_manager.cc:1428] T de0724ba71824a5daaf6f3708b4431b7 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.090s user 0.007s sys 0.000s
I20250626 19:57:51.279444 5410 tablet_bootstrap.cc:492] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:51.283600 4671 raft_consensus.cc:2466] T de0724ba71824a5daaf6f3708b4431b7 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:51.302289 5353 tablet_bootstrap.cc:654] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:51.318383 5366 tablet_bootstrap.cc:492] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:51.364849 5353 tablet_bootstrap.cc:492] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:51.365331 5353 ts_tablet_manager.cc:1397] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.193s user 0.009s sys 0.000s
I20250626 19:57:51.367591 5353 raft_consensus.cc:357] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.368429 5353 raft_consensus.cc:383] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.368734 5353 raft_consensus.cc:738] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:51.372427 5353 consensus_queue.cc:260] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.374413 5353 ts_tablet_manager.cc:1428] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250626 19:57:51.403043 6121 raft_consensus.cc:491] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:51.403544 6121 raft_consensus.cc:513] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.414677 6237 raft_consensus.cc:491] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:51.415485 6237 raft_consensus.cc:513] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.419106 6237 leader_election.cc:290] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:51.420869 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "159a6bf746e14479b3f3584640b63098" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:51.422538 4912 leader_election.cc:343] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:51.428655 5366 tablet_bootstrap.cc:700] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1: Time spent opening tablet: real 0.104s user 0.002s sys 0.001s
I20250626 19:57:51.429296 5353 tablet_bootstrap.cc:492] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:51.429806 6121 leader_election.cc:290] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:51.432024 5366 tablet_bootstrap.cc:654] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:51.435827 5353 tablet_bootstrap.cc:654] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:51.444458 5353 tablet_bootstrap.cc:492] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:51.444849 5353 ts_tablet_manager.cc:1397] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20250626 19:57:51.446976 5410 tablet_bootstrap.cc:654] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:51.447021 5353 raft_consensus.cc:357] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.447607 5353 raft_consensus.cc:383] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.447855 5353 raft_consensus.cc:738] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:51.448413 5353 consensus_queue.cc:260] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.451186 5353 ts_tablet_manager.cc:1428] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250626 19:57:51.452001 5366 tablet_bootstrap.cc:492] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:51.452437 5353 tablet_bootstrap.cc:492] T 4a6058961bc7450ab489b26158e7dd17 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:51.452452 5366 ts_tablet_manager.cc:1397] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.134s user 0.008s sys 0.003s
I20250626 19:57:51.454723 5366 raft_consensus.cc:357] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.456761 5366 raft_consensus.cc:383] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.457095 5366 raft_consensus.cc:738] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:51.457809 5366 consensus_queue.cc:260] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.460017 5366 ts_tablet_manager.cc:1428] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20250626 19:57:51.461232 5366 tablet_bootstrap.cc:492] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:51.468537 5366 tablet_bootstrap.cc:654] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:51.515790 5353 tablet_bootstrap.cc:654] T 4a6058961bc7450ab489b26158e7dd17 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:51.521492 5410 tablet_bootstrap.cc:492] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:51.522003 5410 ts_tablet_manager.cc:1397] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.243s user 0.016s sys 0.001s
I20250626 19:57:51.524629 5410 raft_consensus.cc:357] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.525764 6070 mvcc.cc:204] Tried to move back new op lower bound from 7171964365608230912 to 7171964310484385792. Current Snapshot: MvccSnapshot[applied={T|T < 7171964365608230912 or (T in {7171964365608230912})}]
I20250626 19:57:51.527104 5410 raft_consensus.cc:383] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.527601 5410 raft_consensus.cc:738] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:51.545773 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8282e22bfb8b4d4994fa8b95ef154a68" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:51.533783 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8282e22bfb8b4d4994fa8b95ef154a68" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:51.563421 6121 raft_consensus.cc:491] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:51.544865 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "159a6bf746e14479b3f3584640b63098" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
W20250626 19:57:51.548324 4756 leader_election.cc:343] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:51.565053 5410 consensus_queue.cc:260] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.570349 5410 ts_tablet_manager.cc:1428] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20250626 19:57:51.574723 6242 consensus_queue.cc:1035] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250626 19:57:51.577000 4755 leader_election.cc:343] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:51.578624 6263 raft_consensus.cc:491] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:51.564447 6121 raft_consensus.cc:513] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.579361 6263 raft_consensus.cc:513] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.579963 4755 leader_election.cc:304] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:51.581143 6121 leader_election.cc:290] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:51.581468 6263 leader_election.cc:290] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:51.581595 6121 raft_consensus.cc:2747] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:51.582403 6251 raft_consensus.cc:491] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:51.582981 6251 raft_consensus.cc:513] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.585345 6251 leader_election.cc:290] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:51.585675 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b794133c1a346318e31bffb4ba6effb" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:51.586100 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b794133c1a346318e31bffb4ba6effb" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:51.587152 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54e68eba747e4a2a947e84e53a444d08" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:51.587725 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54e68eba747e4a2a947e84e53a444d08" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:51.588589 6242 raft_consensus.cc:491] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:51.589016 6242 raft_consensus.cc:513] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:51.589576 4755 leader_election.cc:343] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:51.590903 6242 leader_election.cc:290] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:51.591315 4756 leader_election.cc:343] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:51.591773 4756 leader_election.cc:304] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:51.591957 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ea7179912284fef8d2165cdfaf4a21a" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:51.593210 4912 leader_election.cc:343] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:51.590312 4755 leader_election.cc:343] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:51.594058 4756 leader_election.cc:343] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:51.595162 4756 leader_election.cc:304] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:51.596676 6270 raft_consensus.cc:2747] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:51.596719 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ea7179912284fef8d2165cdfaf4a21a" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:51.597338 6270 raft_consensus.cc:2747] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:51.597667 4815 raft_consensus.cc:2466] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:51.599252 4911 leader_election.cc:304] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:51.599912 4819 raft_consensus.cc:2466] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:51.600385 6242 raft_consensus.cc:2802] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:51.600704 6242 raft_consensus.cc:491] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:51.600991 6242 raft_consensus.cc:3058] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:51.602921 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e953f25481a245dd8edd61cd2e6061d1" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:51.603716 4911 leader_election.cc:304] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:51.604207 5410 tablet_bootstrap.cc:492] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:51.604624 6237 raft_consensus.cc:2802] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:51.604931 6237 raft_consensus.cc:491] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:51.605388 6237 raft_consensus.cc:3058] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:51.607846 6242 raft_consensus.cc:513] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:51.611429 4756 leader_election.cc:343] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:51.613085 6242 leader_election.cc:290] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:51.614068 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ea7179912284fef8d2165cdfaf4a21a" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:51.617753 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ea7179912284fef8d2165cdfaf4a21a" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:51.618271 4819 raft_consensus.cc:3058] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:51.617857 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e953f25481a245dd8edd61cd2e6061d1" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:51.619632 4755 leader_election.cc:343] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:51.620018 4755 leader_election.cc:304] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:51.618414 5410 tablet_bootstrap.cc:654] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:51.623162 6251 raft_consensus.cc:2747] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:51.635612 6084 raft_consensus.cc:491] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:51.636158 6084 raft_consensus.cc:513] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.638190 6084 leader_election.cc:290] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:51.640504 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "159a6bf746e14479b3f3584640b63098" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:51.641080 4815 raft_consensus.cc:2466] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:51.642257 4609 leader_election.cc:304] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:51.644059 6084 raft_consensus.cc:2802] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:51.644490 6084 raft_consensus.cc:491] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:51.644874 6084 raft_consensus.cc:3058] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:51.647606 5410 tablet_bootstrap.cc:492] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:51.648015 5410 ts_tablet_manager.cc:1397] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.000s
I20250626 19:57:51.653908 5410 raft_consensus.cc:357] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.656140 4819 raft_consensus.cc:2466] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
W20250626 19:57:51.615463 4912 leader_election.cc:343] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:51.659103 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 43692.9us
I20250626 19:57:51.661409 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "159a6bf746e14479b3f3584640b63098" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:51.663488 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:51.663642 4911 leader_election.cc:304] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:51.664686 6242 raft_consensus.cc:2802] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:51.665122 6242 raft_consensus.cc:695] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:51.622248 6237 raft_consensus.cc:513] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.665856 6242 consensus_queue.cc:237] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.667052 4979 raft_consensus.cc:2391] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:51.642344 5353 tablet_bootstrap.cc:492] T 4a6058961bc7450ab489b26158e7dd17 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:51.668373 5353 ts_tablet_manager.cc:1397] T 4a6058961bc7450ab489b26158e7dd17 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.216s user 0.003s sys 0.007s
I20250626 19:57:51.671029 5353 raft_consensus.cc:357] T 4a6058961bc7450ab489b26158e7dd17 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.670598 6084 raft_consensus.cc:513] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.671686 5353 raft_consensus.cc:383] T 4a6058961bc7450ab489b26158e7dd17 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.671919 5353 raft_consensus.cc:738] T 4a6058961bc7450ab489b26158e7dd17 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:51.672431 5353 consensus_queue.cc:260] T 4a6058961bc7450ab489b26158e7dd17 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.675982 5353 ts_tablet_manager.cc:1428] T 4a6058961bc7450ab489b26158e7dd17 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20250626 19:57:51.676649 6229 consensus_queue.cc:1035] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:51.678644 6237 leader_election.cc:290] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:51.679423 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "159a6bf746e14479b3f3584640b63098" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:51.680088 4979 raft_consensus.cc:2391] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:51.681316 6121 raft_consensus.cc:491] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:51.681591 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "159a6bf746e14479b3f3584640b63098" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:51.682238 4670 raft_consensus.cc:2391] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:51.681838 6121 raft_consensus.cc:513] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.683842 6121 leader_election.cc:290] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:51.690217 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e6136b0d3104ae88c3f4ae88a41de84" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:51.691531 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e6136b0d3104ae88c3f4ae88a41de84" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:51.693089 4756 leader_election.cc:343] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:51.693632 4755 leader_election.cc:343] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:51.694099 4755 leader_election.cc:304] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:51.684904 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "159a6bf746e14479b3f3584640b63098" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:51.695997 5353 tablet_bootstrap.cc:492] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:51.697062 4819 raft_consensus.cc:3058] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:51.695447 6251 raft_consensus.cc:2747] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:51.654557 5410 raft_consensus.cc:383] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.711194 5410 raft_consensus.cc:738] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:51.711937 5410 consensus_queue.cc:260] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.713519 5410 ts_tablet_manager.cc:1428] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.065s user 0.005s sys 0.000s
I20250626 19:57:51.717499 6241 consensus_queue.cc:1035] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:51.722026 5410 tablet_bootstrap.cc:492] T f70c620bf4474e59aa1f86761aca0daa P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:51.729110 5366 tablet_bootstrap.cc:492] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:51.729626 5366 ts_tablet_manager.cc:1397] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.269s user 0.017s sys 0.015s
I20250626 19:57:51.729940 6229 consensus_queue.cc:1035] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:51.738512 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "159a6bf746e14479b3f3584640b63098" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:51.740106 4609 leader_election.cc:304] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:51.740563 5366 raft_consensus.cc:357] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.741366 5366 raft_consensus.cc:383] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.741668 5366 raft_consensus.cc:738] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
W20250626 19:57:51.743000 4609 consensus_peers.cc:487] T f807cf54bbb146ff989762c18410e6bb P 4d3be519a0044a2f921a503412f72e86 -> Peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Couldn't send request to peer 93f885c6802341c889a40e9c20fbfff1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f807cf54bbb146ff989762c18410e6bb. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:51.743183 6229 raft_consensus.cc:2747] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:57:51.744292 6084 leader_election.cc:290] T 159a6bf746e14479b3f3584640b63098 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:51.747200 5353 tablet_bootstrap.cc:654] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:51.778385 4609 consensus_peers.cc:487] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 -> Peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Couldn't send request to peer 93f885c6802341c889a40e9c20fbfff1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc526a05990446a3b3bc973e79511476. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:51.784770 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:51.792001 4819 raft_consensus.cc:2466] T 159a6bf746e14479b3f3584640b63098 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:51.793251 4911 leader_election.cc:304] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:51.794256 5924 raft_consensus.cc:2802] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:51.742394 5366 consensus_queue.cc:260] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.794682 5924 raft_consensus.cc:695] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:51.795445 5924 consensus_queue.cc:237] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.797014 5366 ts_tablet_manager.cc:1428] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.067s user 0.005s sys 0.003s
I20250626 19:57:51.798214 5366 tablet_bootstrap.cc:492] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:51.808594 6296 consensus_queue.cc:1035] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:51.815646 6228 consensus_queue.cc:1035] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:51.823786 5353 tablet_bootstrap.cc:492] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:51.824195 5353 ts_tablet_manager.cc:1397] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.148s user 0.003s sys 0.006s
I20250626 19:57:51.828591 5353 raft_consensus.cc:357] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.829241 5353 raft_consensus.cc:383] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.829476 5353 raft_consensus.cc:738] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:51.836295 5410 tablet_bootstrap.cc:654] T f70c620bf4474e59aa1f86761aca0daa P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:51.836274 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 0f834922c9de46dca48c47748913b11a (table default.Table_6251ce018aaf4ee0948f7c26518cb21f [id=594e3d715a6d42c798c44d20740d2ebb]). Expected version 1 got 0
I20250626 19:57:51.849565 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet e0bcbb044d1549c88e5b83eeb5c709af (table default.Table_573c94e619134cddab6aac4f11c37871 [id=8e37b6f1897b487db295c106a6a8aa3e]). Expected version 1 got 0
I20250626 19:57:51.850921 6296 consensus_queue.cc:1035] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:51.830228 5353 consensus_queue.cc:260] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:51.876392 5353 ts_tablet_manager.cc:1428] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.052s user 0.004s sys 0.002s
I20250626 19:57:51.877259 5353 tablet_bootstrap.cc:492] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:51.877322 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (table default.Table_56852b6e38af4545879ff672cc528975 [id=e3400f897e9944549162854e3f92ac21]). Expected version 1 got 0
I20250626 19:57:51.879289 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 465ed0fe56fb4960a58e92b9b3d594e6 (table default.Table_87f11a0b0a984eccb7c82abd1984bb4d [id=93fc69977dd74f96997f838682b24091]). Expected version 1 got 0
I20250626 19:57:51.907513 5366 tablet_bootstrap.cc:654] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:51.920967 6299 consensus_queue.cc:1035] T fc526a05990446a3b3bc973e79511476 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:51.928646 5410 tablet_bootstrap.cc:492] T f70c620bf4474e59aa1f86761aca0daa P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:51.929088 5410 ts_tablet_manager.cc:1397] T f70c620bf4474e59aa1f86761aca0daa P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.207s user 0.008s sys 0.011s
I20250626 19:57:51.932987 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 6740fcfd3d41487b8a4cb2a6af1e70d4 (table default.Table_8b997467fa764e8fbf273c3b1f2ccb3e [id=760c56838ee140d78aa368a3810d19a8]). Expected version 1 got 0
I20250626 19:57:51.934845 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 27acc0eafbb04375ba1c6a59e958e6f6 (table default.Table_2d1ec663bb8048e0ac15d7025383ef5d [id=30ad5642ec2d422aad5b97698297fbe9]). Expected version 1 got 0
I20250626 19:57:51.957736 6307 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 62ac80df82774b9b87c0be076b5637af)
I20250626 19:57:51.958427 6307 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:51.962930 5836 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 7 candidate_status { last_received { term: 6 index: 290 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
I20250626 19:57:51.964500 5366 tablet_bootstrap.cc:492] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:51.964876 5366 ts_tablet_manager.cc:1397] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.167s user 0.009s sys 0.000s
I20250626 19:57:51.938655 5353 tablet_bootstrap.cc:654] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:51.969755 5366 raft_consensus.cc:357] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.970373 5366 raft_consensus.cc:383] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.970598 5366 raft_consensus.cc:738] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:51.971174 5366 consensus_queue.cc:260] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.972950 5366 ts_tablet_manager.cc:1428] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250626 19:57:51.967186 4407 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 7 candidate_status { last_received { term: 6 index: 290 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:51.980065 4464 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31; no voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:51.981936 6309 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 19:57:51.981848 4390 catalog_manager.cc:5582] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:51.983248 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac]). Expected version 1 got 0
I20250626 19:57:51.994339 6228 raft_consensus.cc:491] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:51.994827 6228 raft_consensus.cc:513] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.996641 6228 leader_election.cc:290] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:51.999044 5410 raft_consensus.cc:357] T f70c620bf4474e59aa1f86761aca0daa P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:51.999392 5924 consensus_queue.cc:1035] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:51.999677 5410 raft_consensus.cc:383] T f70c620bf4474e59aa1f86761aca0daa P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:51.999904 5410 raft_consensus.cc:738] T f70c620bf4474e59aa1f86761aca0daa P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:52.001065 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a6058961bc7450ab489b26158e7dd17" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:52.001617 4973 raft_consensus.cc:2466] T 4a6058961bc7450ab489b26158e7dd17 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:52.002794 4611 leader_election.cc:304] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:52.000456 5410 consensus_queue.cc:260] T f70c620bf4474e59aa1f86761aca0daa P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.004027 6228 raft_consensus.cc:2802] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:52.004506 6228 raft_consensus.cc:491] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:52.004860 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a6058961bc7450ab489b26158e7dd17" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:52.004812 6228 raft_consensus.cc:3058] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:52.005460 4815 raft_consensus.cc:2466] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:52.006031 5924 raft_consensus.cc:491] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:52.006451 5924 raft_consensus.cc:513] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.008513 5924 leader_election.cc:290] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:52.014294 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f70c620bf4474e59aa1f86761aca0daa" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:52.014979 4815 raft_consensus.cc:2466] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:52.015779 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f70c620bf4474e59aa1f86761aca0daa" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:52.016445 4670 raft_consensus.cc:2466] T f70c620bf4474e59aa1f86761aca0daa P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:52.016530 4911 leader_election.cc:304] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:52.017776 6242 raft_consensus.cc:2802] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:52.018191 6242 raft_consensus.cc:491] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:52.018673 6242 raft_consensus.cc:3058] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:52.024943 5410 ts_tablet_manager.cc:1428] T f70c620bf4474e59aa1f86761aca0daa P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.096s user 0.005s sys 0.000s
I20250626 19:57:52.025411 6299 consensus_queue.cc:1035] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:57:52.031306 4464 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.189:45039, user_credentials={real_user=slave}} blocked reactor thread for 51387.9us
I20250626 19:57:52.034900 6229 consensus_queue.cc:1035] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:52.046859 5366 tablet_bootstrap.cc:492] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:52.051450 6263 consensus_queue.cc:1035] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:52.062636 5353 tablet_bootstrap.cc:492] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:52.063072 5353 ts_tablet_manager.cc:1397] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.186s user 0.008s sys 0.007s
I20250626 19:57:52.065141 6307 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:52.065109 5353 raft_consensus.cc:357] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:52.065354 6241 consensus_queue.cc:1035] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:52.065735 5353 raft_consensus.cc:383] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:52.065994 5353 raft_consensus.cc:738] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:52.066609 5353 consensus_queue.cc:260] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:52.076984 5410 tablet_bootstrap.cc:492] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:52.081610 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 34b2919e5f534f4295336a4384be1f64 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:52.082461 5353 ts_tablet_manager.cc:1428] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250626 19:57:52.083397 5353 tablet_bootstrap.cc:492] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:52.082381 4390 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:52.106307 6263 raft_consensus.cc:491] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:52.106804 6263 raft_consensus.cc:513] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.109009 5366 tablet_bootstrap.cc:654] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:52.111793 6263 leader_election.cc:290] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:52.114604 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4aee96ad3454d65b974c867605c4cdc" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:52.115289 4670 raft_consensus.cc:2466] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:52.117542 4755 leader_election.cc:304] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:52.118319 6270 raft_consensus.cc:2802] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 19:57:52.118702 6270 raft_consensus.cc:491] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:52.119143 6270 raft_consensus.cc:3058] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:57:52.131814 6270 raft_consensus.cc:513] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.134903 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4aee96ad3454d65b974c867605c4cdc" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:52.135465 4670 raft_consensus.cc:3058] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:57:52.148145 6121 consensus_queue.cc:1035] T de0724ba71824a5daaf6f3708b4431b7 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:52.152457 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4aee96ad3454d65b974c867605c4cdc" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:52.153983 4973 raft_consensus.cc:2466] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:52.155050 6121 raft_consensus.cc:491] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:52.155491 6121 raft_consensus.cc:513] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.157455 6121 leader_election.cc:290] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:52.158275 5410 tablet_bootstrap.cc:654] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:52.158944 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d668a84475064a3eaa08b0c719471c26" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:52.159821 6270 leader_election.cc:290] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 2 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:52.160935 6228 raft_consensus.cc:513] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:52.161410 4755 leader_election.cc:343] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:52.157444 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4aee96ad3454d65b974c867605c4cdc" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:52.082176 6242 raft_consensus.cc:513] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.178697 4970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d668a84475064a3eaa08b0c719471c26" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:52.181967 4970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a6058961bc7450ab489b26158e7dd17" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:52.182577 4970 raft_consensus.cc:3058] T 4a6058961bc7450ab489b26158e7dd17 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:52.187701 4756 leader_election.cc:343] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:52.188666 4756 leader_election.cc:304] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:52.189376 6121 raft_consensus.cc:2747] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:52.190721 4970 raft_consensus.cc:2466] T 4a6058961bc7450ab489b26158e7dd17 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:52.198398 5353 tablet_bootstrap.cc:654] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:52.201849 5410 tablet_bootstrap.cc:492] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:52.201848 6242 leader_election.cc:290] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:52.202260 5410 ts_tablet_manager.cc:1397] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.125s user 0.009s sys 0.000s
I20250626 19:57:52.203789 6228 leader_election.cc:290] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:52.208010 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a6058961bc7450ab489b26158e7dd17" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:52.208546 4815 raft_consensus.cc:3058] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:52.223923 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f70c620bf4474e59aa1f86761aca0daa" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:52.224514 4816 raft_consensus.cc:3058] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:52.230229 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:52.234917 5410 raft_consensus.cc:357] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.235576 5410 raft_consensus.cc:383] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:52.235821 5410 raft_consensus.cc:738] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:52.242372 4816 raft_consensus.cc:2466] T f70c620bf4474e59aa1f86761aca0daa P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:52.238873 4611 leader_election.cc:304] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:52.248652 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f70c620bf4474e59aa1f86761aca0daa" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:52.250459 4669 raft_consensus.cc:3058] T f70c620bf4474e59aa1f86761aca0daa P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:52.251088 6228 raft_consensus.cc:2802] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:52.251746 6121 raft_consensus.cc:491] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:52.252209 6121 raft_consensus.cc:513] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:52.251910 6228 raft_consensus.cc:695] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:52.254946 6121 leader_election.cc:290] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:52.255173 6228 consensus_queue.cc:237] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.257256 4670 raft_consensus.cc:2466] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 2.
I20250626 19:57:52.255635 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7851e464fcd44ecb82404d6a991db2a2" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:52.257498 4669 raft_consensus.cc:2466] T f70c620bf4474e59aa1f86761aca0daa P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:52.258638 4755 leader_election.cc:304] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:52.258438 5410 consensus_queue.cc:260] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:52.259783 4755 leader_election.cc:343] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:52.260941 6121 raft_consensus.cc:2802] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 19:57:52.261375 5410 ts_tablet_manager.cc:1428] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.059s user 0.006s sys 0.000s
I20250626 19:57:52.261416 6121 raft_consensus.cc:695] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [term 2 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:52.262269 5410 tablet_bootstrap.cc:492] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:52.262368 6121 consensus_queue.cc:237] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.265277 4815 raft_consensus.cc:2466] T 4a6058961bc7450ab489b26158e7dd17 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:52.277341 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:52.287866 4911 leader_election.cc:304] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:52.290043 6048 raft_consensus.cc:2802] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:52.290596 6048 raft_consensus.cc:695] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:52.292152 6048 consensus_queue.cc:237] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.329152 5366 tablet_bootstrap.cc:492] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:52.329558 5366 ts_tablet_manager.cc:1397] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.283s user 0.007s sys 0.007s
W20250626 19:57:52.336140 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 48731.2us
I20250626 19:57:52.338176 5366 raft_consensus.cc:357] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.338863 5366 raft_consensus.cc:383] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:52.339282 5366 raft_consensus.cc:738] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:52.340170 5366 consensus_queue.cc:260] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:52.356079 4383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.190:33823, user_credentials={real_user=slave}} blocked reactor thread for 60026.6us
I20250626 19:57:52.358453 6121 raft_consensus.cc:491] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:52.359609 5353 tablet_bootstrap.cc:492] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:52.359339 6121 raft_consensus.cc:513] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.360044 5410 tablet_bootstrap.cc:654] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:52.360195 5353 ts_tablet_manager.cc:1397] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.277s user 0.014s sys 0.004s
I20250626 19:57:52.362838 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56f319c748374b7cbbf037ef55aecd1b" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:52.364384 4755 leader_election.cc:343] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:52.368357 5366 ts_tablet_manager.cc:1428] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.039s user 0.003s sys 0.004s
I20250626 19:57:52.371717 5353 raft_consensus.cc:357] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.372484 5353 raft_consensus.cc:383] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:52.372774 5353 raft_consensus.cc:738] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:52.373581 5353 consensus_queue.cc:260] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.375489 5366 tablet_bootstrap.cc:492] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:52.376813 5353 ts_tablet_manager.cc:1428] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.016s user 0.004s sys 0.001s
W20250626 19:57:52.388857 4383 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 27acc0eafbb04375ba1c6a59e958e6f6 (table default.Table_2d1ec663bb8048e0ac15d7025383ef5d [id=30ad5642ec2d422aad5b97698297fbe9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 27acc0eafbb04375ba1c6a59e958e6f6 consensus state has no leader
I20250626 19:57:52.399868 5366 tablet_bootstrap.cc:654] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:52.410938 6121 leader_election.cc:290] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:52.410945 4970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56f319c748374b7cbbf037ef55aecd1b" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:52.415069 4756 leader_election.cc:343] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:52.415481 4756 leader_election.cc:304] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:52.410964 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7851e464fcd44ecb82404d6a991db2a2" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:52.416080 4973 raft_consensus.cc:2466] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 0.
I20250626 19:57:52.416440 6121 raft_consensus.cc:2747] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:52.416702 6270 raft_consensus.cc:491] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:52.417111 6270 raft_consensus.cc:513] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.418911 6270 leader_election.cc:290] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:52.421897 5353 tablet_bootstrap.cc:492] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:52.441982 4756 leader_election.cc:304] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:52.442792 6270 raft_consensus.cc:2802] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:52.443146 6270 raft_consensus.cc:491] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:52.443459 6270 raft_consensus.cc:3058] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:52.445075 4651 tablet_service.cc:1468] Processing CreateTablet for tablet 06143664e0de45ac815d1fa2e061aa4b (DEFAULT_TABLE table=default.Table_3129bcf22b744b0d8072d7e291616bc7 [id=3568284a3ae54845a2d044ddf8bd57cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.445312 4959 tablet_service.cc:1468] Processing CreateTablet for tablet 06143664e0de45ac815d1fa2e061aa4b (DEFAULT_TABLE table=default.Table_3129bcf22b744b0d8072d7e291616bc7 [id=3568284a3ae54845a2d044ddf8bd57cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.446339 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06143664e0de45ac815d1fa2e061aa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.446149 4793 tablet_service.cc:1468] Processing CreateTablet for tablet 06143664e0de45ac815d1fa2e061aa4b (DEFAULT_TABLE table=default.Table_3129bcf22b744b0d8072d7e291616bc7 [id=3568284a3ae54845a2d044ddf8bd57cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.447376 4793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06143664e0de45ac815d1fa2e061aa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.452071 5410 tablet_bootstrap.cc:492] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:52.452518 5410 ts_tablet_manager.cc:1397] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.190s user 0.008s sys 0.001s
I20250626 19:57:52.443894 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f54a2d0098446a5b60dc6d07a234bc4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:52.454200 4755 leader_election.cc:343] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:52.455808 5410 raft_consensus.cc:357] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:52.456423 5410 raft_consensus.cc:383] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:52.456657 5410 raft_consensus.cc:738] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:52.456718 4657 tablet_service.cc:1468] Processing CreateTablet for tablet 73cff4612d06483d9611b3014420d3a5 (DEFAULT_TABLE table=default.Table_9f7e67f1042843f89daa2817b09e9fb9 [id=475e8ddd67b447ad8c6b9d2dd59fccf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.457284 5410 consensus_queue.cc:260] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:52.457778 4800 tablet_service.cc:1468] Processing CreateTablet for tablet 73cff4612d06483d9611b3014420d3a5 (DEFAULT_TABLE table=default.Table_9f7e67f1042843f89daa2817b09e9fb9 [id=475e8ddd67b447ad8c6b9d2dd59fccf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.458837 5410 ts_tablet_manager.cc:1428] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250626 19:57:52.458870 4800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73cff4612d06483d9611b3014420d3a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.460561 6270 raft_consensus.cc:513] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:52.463258 4670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7851e464fcd44ecb82404d6a991db2a2" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:52.463446 4952 tablet_service.cc:1468] Processing CreateTablet for tablet 73cff4612d06483d9611b3014420d3a5 (DEFAULT_TABLE table=default.Table_9f7e67f1042843f89daa2817b09e9fb9 [id=475e8ddd67b447ad8c6b9d2dd59fccf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.463761 4670 raft_consensus.cc:3058] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:52.464401 4952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73cff4612d06483d9611b3014420d3a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.488869 4649 tablet_service.cc:1468] Processing CreateTablet for tablet c2ff9841bbbd429b83c123e1ebdabfea (DEFAULT_TABLE table=default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.500814 4796 tablet_service.cc:1468] Processing CreateTablet for tablet c2ff9841bbbd429b83c123e1ebdabfea (DEFAULT_TABLE table=default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.501889 4796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2ff9841bbbd429b83c123e1ebdabfea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.506825 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f54a2d0098446a5b60dc6d07a234bc4" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:52.490624 6270 leader_election.cc:290] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:52.514372 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7851e464fcd44ecb82404d6a991db2a2" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:52.514940 4973 raft_consensus.cc:3058] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:52.516227 5366 tablet_bootstrap.cc:492] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:52.516623 5366 ts_tablet_manager.cc:1397] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.141s user 0.005s sys 0.003s
I20250626 19:57:52.517892 4958 tablet_service.cc:1468] Processing CreateTablet for tablet c2ff9841bbbd429b83c123e1ebdabfea (DEFAULT_TABLE table=default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.518919 4958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2ff9841bbbd429b83c123e1ebdabfea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.519646 4670 raft_consensus.cc:2466] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:52.520684 4755 leader_election.cc:304] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:52.521497 6270 raft_consensus.cc:2802] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:52.521963 6270 raft_consensus.cc:695] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Running, Role: LEADER
I20250626 19:57:52.522799 6270 consensus_queue.cc:237] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:52.523465 4655 tablet_service.cc:1468] Processing CreateTablet for tablet 08045a939eed4ec69b49d68b555bdf80 (DEFAULT_TABLE table=default.Table_ad7eb0792b7444b2891bf198ef117218 [id=8ae02648a2e946c18cbb5f2d52ce375a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.529695 4795 tablet_service.cc:1468] Processing CreateTablet for tablet 08045a939eed4ec69b49d68b555bdf80 (DEFAULT_TABLE table=default.Table_ad7eb0792b7444b2891bf198ef117218 [id=8ae02648a2e946c18cbb5f2d52ce375a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.531596 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08045a939eed4ec69b49d68b555bdf80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.524384 5366 raft_consensus.cc:357] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:52.534554 4801 tablet_service.cc:1468] Processing CreateTablet for tablet 1b0e830b839d4deeac6c7731a0bc9bdb (DEFAULT_TABLE table=default.Table_6635c438f840403e8a1ef43d005d2dad [id=3277338e789949f99dafadfe90a0b043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.535570 4801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0e830b839d4deeac6c7731a0bc9bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.536198 4656 tablet_service.cc:1468] Processing CreateTablet for tablet 1b0e830b839d4deeac6c7731a0bc9bdb (DEFAULT_TABLE table=default.Table_6635c438f840403e8a1ef43d005d2dad [id=3277338e789949f99dafadfe90a0b043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.534983 5366 raft_consensus.cc:383] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:52.537132 5366 raft_consensus.cc:738] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:52.537782 5366 consensus_queue.cc:260] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:52.548317 5366 ts_tablet_manager.cc:1428] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.031s user 0.006s sys 0.001s
W20250626 19:57:52.558508 4756 leader_election.cc:343] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:52.549299 5366 tablet_bootstrap.cc:492] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:52.559121 4756 leader_election.cc:304] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:52.559940 6121 raft_consensus.cc:2747] T 8f54a2d0098446a5b60dc6d07a234bc4 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:52.525198 4973 raft_consensus.cc:2466] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f885c6802341c889a40e9c20fbfff1 in term 1.
I20250626 19:57:52.568838 6318 raft_consensus.cc:491] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:52.532722 4957 tablet_service.cc:1468] Processing CreateTablet for tablet 08045a939eed4ec69b49d68b555bdf80 (DEFAULT_TABLE table=default.Table_ad7eb0792b7444b2891bf198ef117218 [id=8ae02648a2e946c18cbb5f2d52ce375a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.569279 6318 raft_consensus.cc:513] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.569881 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08045a939eed4ec69b49d68b555bdf80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.571141 6318 leader_election.cc:290] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:52.540829 4942 tablet_service.cc:1468] Processing CreateTablet for tablet 1b0e830b839d4deeac6c7731a0bc9bdb (DEFAULT_TABLE table=default.Table_6635c438f840403e8a1ef43d005d2dad [id=3277338e789949f99dafadfe90a0b043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.574340 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ea7179912284fef8d2165cdfaf4a21a" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:52.575145 4816 raft_consensus.cc:2391] T 8ea7179912284fef8d2165cdfaf4a21a P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:52.575345 4651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06143664e0de45ac815d1fa2e061aa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.575922 4657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73cff4612d06483d9611b3014420d3a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.576149 5410 tablet_bootstrap.cc:492] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:52.579200 4649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2ff9841bbbd429b83c123e1ebdabfea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.581807 5410 tablet_bootstrap.cc:654] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:52.583271 4655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08045a939eed4ec69b49d68b555bdf80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.588776 4656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0e830b839d4deeac6c7731a0bc9bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.601873 4942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0e830b839d4deeac6c7731a0bc9bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.613504 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ea7179912284fef8d2165cdfaf4a21a" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
W20250626 19:57:52.614867 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 56413.4us
I20250626 19:57:52.615829 4611 leader_election.cc:304] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:52.616488 6315 raft_consensus.cc:3058] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:52.634922 5410 tablet_bootstrap.cc:492] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:52.635360 5410 ts_tablet_manager.cc:1397] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.000s
I20250626 19:57:52.637439 5410 raft_consensus.cc:357] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.638051 5410 raft_consensus.cc:383] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:52.638273 5410 raft_consensus.cc:738] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:52.639227 6048 consensus_queue.cc:1035] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:52.646530 6315 raft_consensus.cc:2747] T 8ea7179912284fef8d2165cdfaf4a21a P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:52.663496 4655 tablet_service.cc:1468] Processing CreateTablet for tablet ea9a23c8c00443318b10664dc01a652b (DEFAULT_TABLE table=default.Table_a1bd6779cf0e482aba51a93dd289d342 [id=c845dcdf065e4d78a85dabbde3e31fa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.664619 4655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea9a23c8c00443318b10664dc01a652b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.700342 5366 tablet_bootstrap.cc:654] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:52.702795 4959 tablet_service.cc:1468] Processing CreateTablet for tablet ea9a23c8c00443318b10664dc01a652b (DEFAULT_TABLE table=default.Table_a1bd6779cf0e482aba51a93dd289d342 [id=c845dcdf065e4d78a85dabbde3e31fa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.702922 5410 consensus_queue.cc:260] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.704032 4959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea9a23c8c00443318b10664dc01a652b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.705041 4795 tablet_service.cc:1468] Processing CreateTablet for tablet ea9a23c8c00443318b10664dc01a652b (DEFAULT_TABLE table=default.Table_a1bd6779cf0e482aba51a93dd289d342 [id=c845dcdf065e4d78a85dabbde3e31fa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:57:52.710680 4282 master-stress-test.cc:577] Killing test thread 1/10
I20250626 19:57:52.720670 4795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea9a23c8c00443318b10664dc01a652b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:57:52.729791 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:52.732223 5410 ts_tablet_manager.cc:1428] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.097s user 0.005s sys 0.000s
I20250626 19:57:52.754242 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (table default.Table_56852b6e38af4545879ff672cc528975 [id=e3400f897e9944549162854e3f92ac21]). Expected version 1 got 0
I20250626 19:57:52.760915 5353 tablet_bootstrap.cc:654] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:52.739437 5410 tablet_bootstrap.cc:492] T d668a84475064a3eaa08b0c719471c26 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:52.766399 6241 consensus_queue.cc:1035] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:52.796272 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 27acc0eafbb04375ba1c6a59e958e6f6 (table default.Table_2d1ec663bb8048e0ac15d7025383ef5d [id=30ad5642ec2d422aad5b97698297fbe9]). Expected version 1 got 0
I20250626 19:57:52.797191 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac]). Expected version 1 got 0
I20250626 19:57:52.805104 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 34b2919e5f534f4295336a4384be1f64 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:52.806334 4395 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:52.808540 4395 catalog_manager.cc:5582] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:52.812863 5353 tablet_bootstrap.cc:492] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:52.813295 5353 ts_tablet_manager.cc:1397] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.392s user 0.009s sys 0.005s
I20250626 19:57:52.813628 5366 tablet_bootstrap.cc:492] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:52.814028 5924 consensus_queue.cc:1035] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:52.815429 5366 ts_tablet_manager.cc:1397] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.266s user 0.005s sys 0.007s
I20250626 19:57:52.816187 5353 raft_consensus.cc:357] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.816788 5353 raft_consensus.cc:383] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:52.817037 5353 raft_consensus.cc:738] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:52.818012 5366 raft_consensus.cc:357] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.818686 5366 raft_consensus.cc:383] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:52.819125 5366 raft_consensus.cc:738] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:52.819216 4391 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet d8be634cb36d40dca3f9338dab93ad2c
I20250626 19:57:52.820690 4391 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34796 (request call id 191) took 4547 ms. Trace:
I20250626 19:57:52.821075 5366 consensus_queue.cc:260] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.824061 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet d8be634cb36d40dca3f9338dab93ad2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:49 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:52.824390 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet d8be634cb36d40dca3f9338dab93ad2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:49 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:52.825827 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet d8be634cb36d40dca3f9338dab93ad2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:49 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:52.821239 4391 rpcz_store.cc:276] 0626 19:57:48.273004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:49.024914 (+751910us) service_pool.cc:225] Handling call
0626 19:57:49.639047 (+614133us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:49.640030 (+ 983us) catalog_manager.cc:2790] Modifying in-memory table state
0626 19:57:49.640156 (+ 126us) catalog_manager.cc:2799] Locking tablets
0626 19:57:49.647556 (+ 7400us) catalog_manager.cc:2813] Removing table and tablets from system table
0626 19:57:50.011324 (+363768us) catalog_manager.cc:2831] Removing table from by-name map
0626 19:57:50.011531 (+ 207us) catalog_manager.cc:2847] Committing in-memory state
0626 19:57:52.819091 (+2807560us) catalog_manager.cc:2851] Aborting table tasks
0626 19:57:52.820603 (+ 1512us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:49.649867 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:49.650311 (+ 444us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:49.650336 (+ 25us) write_op.cc:435] Acquired schema lock
0626 19:57:49.650348 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:49.650674 (+ 326us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:49.650731 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:49.650788 (+ 57us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:49.651060 (+ 272us) tablet.cc:699] Row locks acquired
0626 19:57:49.651077 (+ 17us) write_op.cc:260] PREPARE: finished
0626 19:57:49.651209 (+ 132us) write_op.cc:270] Start()
0626 19:57:49.651298 (+ 89us) write_op.cc:276] Timestamp: P: 1750967869651182 usec, L: 0
0626 19:57:49.651316 (+ 18us) op_driver.cc:348] REPLICATION: starting
0626 19:57:49.651818 (+ 502us) log.cc:844] Serialized 728 byte log entry
0626 19:57:49.979070 (+327252us) op_driver.cc:464] REPLICATION: finished
0626 19:57:49.992376 (+ 13306us) write_op.cc:301] APPLY: starting
0626 19:57:49.992448 (+ 72us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:49.992552 (+ 104us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:49.992650 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:49.992889 (+ 239us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:49.993047 (+ 158us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:49.993095 (+ 48us) write_op.cc:312] APPLY: finished
0626 19:57:49.993290 (+ 195us) log.cc:844] Serialized 29 byte log entry
0626 19:57:50.010953 (+ 17663us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:50.011027 (+ 74us) write_op.cc:454] Released schema lock
0626 19:57:50.011064 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:50.011739 (+ 675us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:50.011769 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":13125,"apply.run_cpu_time_us":1526,"apply.run_wall_time_us":19329,"num_ops":2,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":2264,"prepare.run_wall_time_us":2263,"raft.queue_time_us":199,"raft.run_cpu_time_us":675,"raft.run_wall_time_us":674,"replication_time_us":327656,"thread_start_us":14579,"threads_started":1}]]}
W20250626 19:57:52.832046 4391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 87) took 2902 ms (client timeout 1000 ms). Trace:
W20250626 19:57:52.832476 4391 rpcz_store.cc:269] 0626 19:57:49.929170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:52.831856 (+2902686us) service_pool.cc:212] Skipping call since client already timed out
0626 19:57:52.831966 (+ 110us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250626 19:57:52.835891 4391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 82) took 2845 ms (client timeout 1000 ms). Trace:
W20250626 19:57:52.836331 4391 rpcz_store.cc:269] 0626 19:57:49.990540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:52.835685 (+2845145us) service_pool.cc:212] Skipping call since client already timed out
0626 19:57:52.835821 (+ 136us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250626 19:57:52.843333 4396 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1b794133c1a346318e31bffb4ba6effb consensus state has no leader
I20250626 19:57:52.844002 4396 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34796 (request call id 192) took 4413 ms. Trace:
I20250626 19:57:52.849431 5353 consensus_queue.cc:260] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.850080 6121 raft_consensus.cc:491] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:52.850545 6121 raft_consensus.cc:513] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:52.851590 6353 tablet_replica.cc:331] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:52.844376 4396 rpcz_store.cc:276] 0626 19:57:48.430705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:49.663088 (+1232383us) service_pool.cc:225] Handling call
0626 19:57:49.664200 (+ 1112us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:49.666357 (+ 2157us) catalog_manager.cc:3683] Apply alter schema
0626 19:57:49.667135 (+ 778us) catalog_manager.cc:3715] Acquired catalog manager lock
0626 19:57:49.667406 (+ 271us) catalog_manager.cc:3873] Updating metadata on disk
0626 19:57:50.049668 (+382262us) catalog_manager.cc:3904] Committing alterations to in-memory state
0626 19:57:52.843931 (+2794263us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:49.668791 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:49.669229 (+ 438us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:49.669254 (+ 25us) write_op.cc:435] Acquired schema lock
0626 19:57:49.669271 (+ 17us) tablet.cc:660] Decoding operations
0626 19:57:49.669549 (+ 278us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:49.669586 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:49.669611 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:49.669766 (+ 155us) tablet.cc:699] Row locks acquired
0626 19:57:49.669782 (+ 16us) write_op.cc:260] PREPARE: finished
0626 19:57:49.669902 (+ 120us) write_op.cc:270] Start()
0626 19:57:49.670007 (+ 105us) write_op.cc:276] Timestamp: P: 1750967869669886 usec, L: 0
0626 19:57:49.670027 (+ 20us) op_driver.cc:348] REPLICATION: starting
0626 19:57:49.670557 (+ 530us) log.cc:844] Serialized 405 byte log entry
0626 19:57:50.026867 (+356310us) op_driver.cc:464] REPLICATION: finished
0626 19:57:50.027597 (+ 730us) write_op.cc:301] APPLY: starting
0626 19:57:50.027649 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:50.027744 (+ 95us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:50.027825 (+ 81us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:50.027967 (+ 142us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:50.028086 (+ 119us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:50.028125 (+ 39us) write_op.cc:312] APPLY: finished
0626 19:57:50.028314 (+ 189us) log.cc:844] Serialized 23 byte log entry
0626 19:57:50.029534 (+ 1220us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:50.029593 (+ 59us) write_op.cc:454] Released schema lock
0626 19:57:50.029626 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:50.049980 (+ 20354us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:50.050005 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":572,"apply.run_cpu_time_us":1206,"apply.run_wall_time_us":22350,"num_ops":1,"prepare.queue_time_us":115,"prepare.run_cpu_time_us":1970,"prepare.run_wall_time_us":1970,"replication_time_us":356698}]]}
W20250626 19:57:52.852612 4396 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 74) took 2185 ms (client timeout 1000 ms). Trace:
W20250626 19:57:52.852913 4396 rpcz_store.cc:269] 0626 19:57:50.667365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:52.852449 (+2185084us) service_pool.cc:212] Skipping call since client already timed out
0626 19:57:52.852557 (+ 108us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250626 19:57:52.853302 6121 leader_election.cc:290] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
W20250626 19:57:52.854126 4396 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 88) took 1629 ms (client timeout 1000 ms). Trace:
W20250626 19:57:52.854418 4396 rpcz_store.cc:269] 0626 19:57:51.224301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:52.853982 (+1629681us) service_pool.cc:212] Skipping call since client already timed out
0626 19:57:52.854076 (+ 94us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250626 19:57:52.855417 6353 raft_consensus.cc:2241] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:52.855985 6353 raft_consensus.cc:2270] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:52.858623 6353 ts_tablet_manager.cc:1905] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:52.859997 4396 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 83) took 1625 ms (client timeout 1000 ms). Trace:
W20250626 19:57:52.860320 4396 rpcz_store.cc:269] 0626 19:57:51.234518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:52.859822 (+1625304us) service_pool.cc:212] Skipping call since client already timed out
0626 19:57:52.859935 (+ 113us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250626 19:57:52.865929 4393 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 7851e464fcd44ecb82404d6a991db2a2
I20250626 19:57:52.868963 4393 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34796 (request call id 190) took 4686 ms. Trace:
I20250626 19:57:52.872161 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 7851e464fcd44ecb82404d6a991db2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:48 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:52.869330 4393 rpcz_store.cc:276] 0626 19:57:48.182105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:48.969208 (+787103us) service_pool.cc:225] Handling call
0626 19:57:48.976554 (+ 7346us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:48.982484 (+ 5930us) catalog_manager.cc:2790] Modifying in-memory table state
0626 19:57:48.984191 (+ 1707us) catalog_manager.cc:2799] Locking tablets
0626 19:57:49.029145 (+ 44954us) catalog_manager.cc:2813] Removing table and tablets from system table
0626 19:57:49.089812 (+ 60667us) catalog_manager.cc:2831] Removing table from by-name map
0626 19:57:49.485300 (+395488us) catalog_manager.cc:2847] Committing in-memory state
0626 19:57:52.865828 (+3380528us) catalog_manager.cc:2851] Aborting table tasks
0626 19:57:52.868905 (+ 3077us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:49.045857 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:49.047873 (+ 2016us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:49.048085 (+ 212us) write_op.cc:435] Acquired schema lock
0626 19:57:49.048280 (+ 195us) tablet.cc:660] Decoding operations
0626 19:57:49.051010 (+ 2730us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:49.051240 (+ 230us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:49.051449 (+ 209us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:49.053987 (+ 2538us) tablet.cc:699] Row locks acquired
0626 19:57:49.054182 (+ 195us) write_op.cc:260] PREPARE: finished
0626 19:57:49.055747 (+ 1565us) write_op.cc:270] Start()
0626 19:57:49.056500 (+ 753us) write_op.cc:276] Timestamp: P: 1750967869054995 usec, L: 0
0626 19:57:49.056523 (+ 23us) op_driver.cc:348] REPLICATION: starting
0626 19:57:49.061988 (+ 5465us) log.cc:844] Serialized 759 byte log entry
0626 19:57:49.074055 (+ 12067us) op_driver.cc:464] REPLICATION: finished
0626 19:57:49.078397 (+ 4342us) write_op.cc:301] APPLY: starting
0626 19:57:49.078455 (+ 58us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:49.078557 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:49.078667 (+ 110us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:49.079027 (+ 360us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:49.079174 (+ 147us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:57:49.079215 (+ 41us) write_op.cc:312] APPLY: finished
0626 19:57:49.079407 (+ 192us) log.cc:844] Serialized 29 byte log entry
0626 19:57:49.080006 (+ 599us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.080075 (+ 69us) write_op.cc:454] Released schema lock
0626 19:57:49.080107 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:49.273707 (+193600us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b180007c660. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe791479af1 00007fe791486441 00007fe78885b60f 00007fe78885d8b1 00007fe78885d869 00007fe78885d7f9 00007fe78885d7c1 00007fe78885cabd 000000000059bcd4 000000000059bb09 00007fe78883e636
0626 19:57:49.273797 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:49.273826 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4160,"apply.run_cpu_time_us":1407,"apply.run_wall_time_us":1921,"num_ops":2,"prepare.queue_time_us":4516,"prepare.run_cpu_time_us":5403,"prepare.run_wall_time_us":85227,"raft.queue_time_us":1206,"raft.run_cpu_time_us":678,"raft.run_wall_time_us":6583,"replication_time_us":17258,"spinlock_wait_cycles":55271680}]]}
I20250626 19:57:52.876662 6353 ts_tablet_manager.cc:1918] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:52.877108 6353 log.cc:1199] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/d8be634cb36d40dca3f9338dab93ad2c
I20250626 19:57:52.878031 6353 ts_tablet_manager.cc:1939] T d8be634cb36d40dca3f9338dab93ad2c P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:52.880967 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet d8be634cb36d40dca3f9338dab93ad2c (table default.Table_6dc13f5ceb674e40919de2ca36d9427a [id=160181772a0a40ebac8a3fe697667b6e]) successfully deleted
I20250626 19:57:52.826653 5410 tablet_bootstrap.cc:654] T d668a84475064a3eaa08b0c719471c26 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:52.891129 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet 7851e464fcd44ecb82404d6a991db2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:48 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:52.892346 4394 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34796 (request call id 193) took 4326 ms. Trace:
I20250626 19:57:52.894184 5366 ts_tablet_manager.cc:1428] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.078s user 0.005s sys 0.000s
I20250626 19:57:52.895104 5366 tablet_bootstrap.cc:492] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:52.892689 4394 rpcz_store.cc:276] 0626 19:57:48.566171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:49.720997 (+1154826us) service_pool.cc:225] Handling call
0626 19:57:49.722040 (+ 1043us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:57:49.722782 (+ 742us) catalog_manager.cc:3683] Apply alter schema
0626 19:57:49.723346 (+ 564us) catalog_manager.cc:3715] Acquired catalog manager lock
0626 19:57:49.723536 (+ 190us) catalog_manager.cc:3873] Updating metadata on disk
0626 19:57:50.057034 (+333498us) catalog_manager.cc:3904] Committing alterations to in-memory state
0626 19:57:52.892289 (+2835255us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:49.724739 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:49.725192 (+ 453us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:49.725228 (+ 36us) write_op.cc:435] Acquired schema lock
0626 19:57:49.725245 (+ 17us) tablet.cc:660] Decoding operations
0626 19:57:49.725567 (+ 322us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:49.725607 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:49.725635 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:49.725793 (+ 158us) tablet.cc:699] Row locks acquired
0626 19:57:49.725807 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:57:49.725930 (+ 123us) write_op.cc:270] Start()
0626 19:57:49.726031 (+ 101us) write_op.cc:276] Timestamp: P: 1750967869725937 usec, L: 0
0626 19:57:49.726052 (+ 21us) op_driver.cc:348] REPLICATION: starting
0626 19:57:49.726611 (+ 559us) log.cc:844] Serialized 405 byte log entry
0626 19:57:50.027106 (+300495us) op_driver.cc:464] REPLICATION: finished
0626 19:57:50.050248 (+ 23142us) write_op.cc:301] APPLY: starting
0626 19:57:50.050297 (+ 49us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:50.050382 (+ 85us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:50.050470 (+ 88us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:50.050622 (+ 152us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:50.050745 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:50.055471 (+ 4726us) write_op.cc:312] APPLY: finished
0626 19:57:50.055645 (+ 174us) log.cc:844] Serialized 23 byte log entry
0626 19:57:50.056639 (+ 994us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:50.056698 (+ 59us) write_op.cc:454] Released schema lock
0626 19:57:50.056727 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":23002,"apply.run_cpu_time_us":1211,"apply.run_wall_time_us":6674,"num_ops":1,"prepare.queue_time_us":111,"prepare.run_cpu_time_us":2143,"prepare.run_wall_time_us":2143,"replication_time_us":300977,"thread_start_us":101571,"threads_started":1}]]}
I20250626 19:57:52.855046 5924 raft_consensus.cc:491] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:52.902992 5924 raft_consensus.cc:513] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:52.904664 5924 leader_election.cc:290] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:52.858933 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d0a30ee16d043f9837dce612dae670a" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:52.883067 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet 7851e464fcd44ecb82404d6a991db2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:48 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:52.892580 6352 tablet_replica.cc:331] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:52.911339 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8282e22bfb8b4d4994fa8b95ef154a68" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:52.911932 4816 raft_consensus.cc:2466] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:52.912693 5353 ts_tablet_manager.cc:1428] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.099s user 0.005s sys 0.000s
W20250626 19:57:52.913578 4756 leader_election.cc:343] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:52.912899 4911 leader_election.cc:304] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:52.920081 5924 raft_consensus.cc:2802] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:52.920380 5924 raft_consensus.cc:491] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:52.920625 5924 raft_consensus.cc:3058] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:52.920568 4611 consensus_peers.cc:487] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 -> Peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Couldn't send request to peer 5f8fb6d535ea42359fe34478df9e00a6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:52.927470 4392 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: default.Table_5fe142a9bd464f8ea2a070dda8bb61e9 [id=ab0bf254569748ba8d31931d4bfb105d] alter complete (version 1)
I20250626 19:57:52.926733 5924 raft_consensus.cc:513] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:52.930439 5353 tablet_bootstrap.cc:492] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:52.933635 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8282e22bfb8b4d4994fa8b95ef154a68" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:52.934137 4816 raft_consensus.cc:3058] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:52.945827 6353 tablet_replica.cc:331] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:52.952215 5366 tablet_bootstrap.cc:654] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:52.955135 6353 raft_consensus.cc:2241] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:52.956043 6353 pending_rounds.cc:70] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1: Trying to abort 1 pending ops.
I20250626 19:57:52.956321 6353 pending_rounds.cc:77] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964405881774080 op_type: NO_OP noop_request { }
I20250626 19:57:52.956774 6353 raft_consensus.cc:2887] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:57:52.957178 6353 raft_consensus.cc:2270] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:52.962790 6353 ts_tablet_manager.cc:1905] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:52.951397 4391 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_636a5548627040c58c26fc1b97b0c366"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:52.972101 4816 raft_consensus.cc:2466] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:52.933091 5924 leader_election.cc:290] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:52.972362 4391 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_636a5548627040c58c26fc1b97b0c366 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:52.973098 4911 leader_election.cc:304] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:52.974231 6239 raft_consensus.cc:2802] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:52.974509 5924 raft_consensus.cc:491] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:52.974676 6239 raft_consensus.cc:695] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:52.976449 6239 consensus_queue.cc:237] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:52.980152 5366 tablet_bootstrap.cc:492] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:52.980583 5366 ts_tablet_manager.cc:1397] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.086s user 0.010s sys 0.000s
I20250626 19:57:52.974912 5924 raft_consensus.cc:513] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:52.991170 4668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d0a30ee16d043f9837dce612dae670a" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:52.991573 5366 raft_consensus.cc:357] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:52.992005 5924 leader_election.cc:290] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:52.992168 5366 raft_consensus.cc:383] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:52.992125 4675 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8282e22bfb8b4d4994fa8b95ef154a68" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:52.992398 5366 raft_consensus.cc:738] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:52.992962 5366 consensus_queue.cc:260] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
W20250626 19:57:52.995126 4912 leader_election.cc:343] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:52.995715 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8282e22bfb8b4d4994fa8b95ef154a68" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:52.996347 6048 consensus_queue.cc:1035] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:52.998095 4673 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d668a84475064a3eaa08b0c719471c26" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:52.999689 4755 leader_election.cc:343] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:53.000207 4755 leader_election.cc:304] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:53.001066 6352 raft_consensus.cc:2241] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:53.001576 6352 raft_consensus.cc:2270] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:53.003932 6352 ts_tablet_manager.cc:1905] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:57:53.017071 4912 leader_election.cc:343] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:53.019078 4609 consensus_peers.cc:487] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 -> Peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Couldn't send request to peer 93f885c6802341c889a40e9c20fbfff1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8be634cb36d40dca3f9338dab93ad2c. This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:53.027897 4912 leader_election.cc:343] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:53.030246 6351 tablet_replica.cc:331] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:53.031250 6351 raft_consensus.cc:2241] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:53.032927 6364 tablet.cc:1722] T 5c113e03594b428a979435618fe373c6 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:53.039494 6121 raft_consensus.cc:2747] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:53.041738 5366 ts_tablet_manager.cc:1428] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.061s user 0.005s sys 0.000s
I20250626 19:57:53.048324 5410 tablet_bootstrap.cc:492] T d668a84475064a3eaa08b0c719471c26 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:53.048749 5410 ts_tablet_manager.cc:1397] T d668a84475064a3eaa08b0c719471c26 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.309s user 0.016s sys 0.006s
I20250626 19:57:53.050601 5353 tablet_bootstrap.cc:654] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:53.052758 4819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d668a84475064a3eaa08b0c719471c26" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:53.053143 5366 tablet_bootstrap.cc:492] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:53.053288 4819 raft_consensus.cc:2466] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:53.054391 5410 raft_consensus.cc:357] T d668a84475064a3eaa08b0c719471c26 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.056879 6359 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 62ac80df82774b9b87c0be076b5637af)
I20250626 19:57:53.057390 6359 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 33823 } } peers { permanent_uuid: "62ac80df82774b9b87c0be076b5637af" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 45039 } } peers { permanent_uuid: "ae949d1fcdf54e87930871ac0ae7da31" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 40801 } }
I20250626 19:57:53.064013 5836 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 7 candidate_status { last_received { term: 6 index: 292 } } ignore_live_leader: false dest_uuid: "e6c58006b6be41c1bd6bdb5eb010d360" is_pre_election: true
W20250626 19:57:53.069757 6358 log.cc:927] Time spent T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20250626 19:57:53.079613 6351 raft_consensus.cc:2270] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:53.081913 6369 tablet.cc:1722] T 5c113e03594b428a979435618fe373c6 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:53.085553 6362 tablet.cc:1722] T 8f1a296c114b4149bc694f6b08a1f2e9 P 4d3be519a0044a2f921a503412f72e86: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:53.061112 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:53.072770 4911 leader_election.cc:304] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:53.091722 6241 raft_consensus.cc:2802] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:53.092142 6241 raft_consensus.cc:491] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:53.092507 6241 raft_consensus.cc:3058] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:53.110164 6239 raft_consensus.cc:491] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:53.110841 6352 ts_tablet_manager.cc:1918] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:53.114801 6359 leader_election.cc:290] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e6c58006b6be41c1bd6bdb5eb010d360 (127.4.46.190:33823), 62ac80df82774b9b87c0be076b5637af (127.4.46.189:45039)
I20250626 19:57:53.055032 5410 raft_consensus.cc:383] T d668a84475064a3eaa08b0c719471c26 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:53.115316 5410 raft_consensus.cc:738] T d668a84475064a3eaa08b0c719471c26 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:53.116041 5410 consensus_queue.cc:260] T d668a84475064a3eaa08b0c719471c26 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.118193 6370 tablet.cc:1722] T 8f1a296c114b4149bc694f6b08a1f2e9 P 5f8fb6d535ea42359fe34478df9e00a6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:53.118640 5410 ts_tablet_manager.cc:1428] T d668a84475064a3eaa08b0c719471c26 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.070s user 0.006s sys 0.000s
I20250626 19:57:53.123878 6241 raft_consensus.cc:513] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.110725 6239 raft_consensus.cc:513] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.124333 6315 raft_consensus.cc:491] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:53.126883 6239 leader_election.cc:290] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:53.127068 6315 raft_consensus.cc:513] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.127419 6352 log.cc:1199] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/d8be634cb36d40dca3f9338dab93ad2c
I20250626 19:57:53.127482 5366 tablet_bootstrap.cc:654] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:53.128502 6352 ts_tablet_manager.cc:1939] T d8be634cb36d40dca3f9338dab93ad2c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:53.128809 6315 leader_election.cc:290] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:53.130090 6352 tablet_replica.cc:331] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:53.130697 5410 tablet_bootstrap.cc:492] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:53.131026 6352 raft_consensus.cc:2241] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:53.131486 6352 raft_consensus.cc:2270] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:53.135447 4673 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e6136b0d3104ae88c3f4ae88a41de84" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:53.135779 4609 leader_election.cc:343] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Not found: Tablet not found: e94a73b50b564a97bbee0f5015bb8805
I20250626 19:57:53.137887 5410 tablet_bootstrap.cc:654] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:53.116444 6351 ts_tablet_manager.cc:1905] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:53.145011 5353 tablet_bootstrap.cc:492] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:53.145403 5353 ts_tablet_manager.cc:1397] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.215s user 0.007s sys 0.003s
W20250626 19:57:53.147305 4912 leader_election.cc:343] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:53.155155 4611 leader_election.cc:343] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Not found: Tablet not found: e94a73b50b564a97bbee0f5015bb8805
I20250626 19:57:53.155558 4611 leader_election.cc:304] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86; no voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:53.156145 6241 leader_election.cc:290] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:53.156447 6315 raft_consensus.cc:2747] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:53.157464 4673 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d668a84475064a3eaa08b0c719471c26" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:53.158035 4673 raft_consensus.cc:3058] T d668a84475064a3eaa08b0c719471c26 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:53.163710 6251 raft_consensus.cc:491] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:53.165104 6251 raft_consensus.cc:513] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:53.169282 4673 raft_consensus.cc:2466] T d668a84475064a3eaa08b0c719471c26 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:53.169623 6251 leader_election.cc:290] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:53.170257 4912 leader_election.cc:304] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6; no voters:
I20250626 19:57:53.171200 6239 raft_consensus.cc:2802] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:53.171777 6239 raft_consensus.cc:695] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:53.172602 4674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34b2919e5f534f4295336a4384be1f64" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:53.173079 4676 raft_consensus.cc:1273] T e4aee96ad3454d65b974c867605c4cdc P 4d3be519a0044a2f921a503412f72e86 [term 2 FOLLOWER]: Refusing update from remote peer 93f885c6802341c889a40e9c20fbfff1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250626 19:57:53.172737 6239 consensus_queue.cc:237] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:53.175011 4755 leader_election.cc:343] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:53.175895 6121 consensus_queue.cc:1035] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:53.183727 5353 raft_consensus.cc:357] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.184542 5353 raft_consensus.cc:383] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:53.184675 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e6136b0d3104ae88c3f4ae88a41de84" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:53.184820 5353 raft_consensus.cc:738] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:53.185216 4821 raft_consensus.cc:2466] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:53.185425 5353 consensus_queue.cc:260] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.186601 6352 ts_tablet_manager.cc:1905] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:53.185210 6353 ts_tablet_manager.cc:1918] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:53.203145 6353 log.cc:1199] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/7851e464fcd44ecb82404d6a991db2a2
I20250626 19:57:53.203976 6353 ts_tablet_manager.cc:1939] T 7851e464fcd44ecb82404d6a991db2a2 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:53.205940 6372 tablet.cc:1722] T 8f1a296c114b4149bc694f6b08a1f2e9 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:53.217466 5353 ts_tablet_manager.cc:1428] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.072s user 0.006s sys 0.001s
I20250626 19:57:53.218564 5353 tablet_bootstrap.cc:492] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:53.238236 5410 tablet_bootstrap.cc:492] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:53.238674 5410 ts_tablet_manager.cc:1397] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.108s user 0.008s sys 0.002s
I20250626 19:57:53.241259 5410 raft_consensus.cc:357] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.243278 5410 raft_consensus.cc:383] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:53.243548 5410 raft_consensus.cc:738] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:53.250404 5410 consensus_queue.cc:260] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.263788 6375 tablet.cc:1722] T 5c113e03594b428a979435618fe373c6 P 93f885c6802341c889a40e9c20fbfff1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:57:53.264642 5353 tablet_bootstrap.cc:654] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:53.269930 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 7851e464fcd44ecb82404d6a991db2a2 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]) successfully deleted
I20250626 19:57:53.289269 6352 ts_tablet_manager.cc:1918] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:53.289753 6352 log.cc:1199] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/7851e464fcd44ecb82404d6a991db2a2
I20250626 19:57:53.290401 6352 ts_tablet_manager.cc:1939] T 7851e464fcd44ecb82404d6a991db2a2 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:53.293656 5410 ts_tablet_manager.cc:1428] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.054s user 0.005s sys 0.000s
I20250626 19:57:53.294529 5410 tablet_bootstrap.cc:492] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:53.269017 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d668a84475064a3eaa08b0c719471c26" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:53.295094 4821 raft_consensus.cc:3058] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:53.272356 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet d8be634cb36d40dca3f9338dab93ad2c (table default.Table_6dc13f5ceb674e40919de2ca36d9427a [id=160181772a0a40ebac8a3fe697667b6e]) successfully deleted
I20250626 19:57:53.316635 6241 consensus_queue.cc:1035] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:57:53.323582 4407 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae949d1fcdf54e87930871ac0ae7da31" candidate_term: 7 candidate_status { last_received { term: 6 index: 292 } } ignore_live_leader: false dest_uuid: "62ac80df82774b9b87c0be076b5637af" is_pre_election: true
I20250626 19:57:53.325877 4464 leader_election.cc:304] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae949d1fcdf54e87930871ac0ae7da31; no voters: 62ac80df82774b9b87c0be076b5637af, e6c58006b6be41c1bd6bdb5eb010d360
I20250626 19:57:53.326926 6359 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ae949d1fcdf54e87930871ac0ae7da31 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 19:57:53.329505 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cc96a3e9c174cbca0196c5a1b27455c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:40742
W20250626 19:57:53.332513 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 3cc96a3e9c174cbca0196c5a1b27455c because the tablet was not found. No further retry: Not found: Tablet not found: 3cc96a3e9c174cbca0196c5a1b27455c
I20250626 19:57:53.338873 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 9172cb6576b240f6b25c3b1267618e6c with delete_type TABLET_DATA_DELETED (replaced by tablet 290232b0933b451c9d7744165bd72796) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:53.339586 5410 tablet_bootstrap.cc:654] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:53.340461 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 9172cb6576b240f6b25c3b1267618e6c because the tablet was not found. No further retry: Not found: Tablet not found: 9172cb6576b240f6b25c3b1267618e6c
I20250626 19:57:53.344545 5366 tablet_bootstrap.cc:492] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:53.344957 5366 ts_tablet_manager.cc:1397] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.292s user 0.013s sys 0.024s
I20250626 19:57:53.347137 5366 raft_consensus.cc:357] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.347862 5366 raft_consensus.cc:383] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:53.347963 4821 raft_consensus.cc:2466] T d668a84475064a3eaa08b0c719471c26 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:53.348124 5366 raft_consensus.cc:738] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:53.349857 5366 consensus_queue.cc:260] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.352195 5366 ts_tablet_manager.cc:1428] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250626 19:57:53.356973 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 7851e464fcd44ecb82404d6a991db2a2 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]) successfully deleted
I20250626 19:57:53.357861 5366 tablet_bootstrap.cc:492] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:53.366178 6351 ts_tablet_manager.cc:1918] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:53.380328 6351 log.cc:1199] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/d8be634cb36d40dca3f9338dab93ad2c
I20250626 19:57:53.382027 5366 tablet_bootstrap.cc:654] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:53.380780 5410 tablet_bootstrap.cc:492] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:53.382637 5410 ts_tablet_manager.cc:1397] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.088s user 0.012s sys 0.000s
I20250626 19:57:53.385006 5410 raft_consensus.cc:357] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.385836 5410 raft_consensus.cc:383] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:53.386082 5410 raft_consensus.cc:738] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:53.386253 6351 ts_tablet_manager.cc:1939] T d8be634cb36d40dca3f9338dab93ad2c P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:53.391602 6351 tablet_replica.cc:331] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:53.409826 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet d8be634cb36d40dca3f9338dab93ad2c (table default.Table_6dc13f5ceb674e40919de2ca36d9427a [id=160181772a0a40ebac8a3fe697667b6e]) successfully deleted
I20250626 19:57:53.410910 5366 tablet_bootstrap.cc:492] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:53.411548 5366 ts_tablet_manager.cc:1397] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.004s
I20250626 19:57:53.413930 5366 raft_consensus.cc:357] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.389501 5410 consensus_queue.cc:260] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.400833 6228 consensus_queue.cc:1035] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:53.415678 6315 raft_consensus.cc:491] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:53.416081 6318 consensus_queue.cc:1035] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:53.417714 5410 ts_tablet_manager.cc:1428] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20250626 19:57:53.418604 5410 tablet_bootstrap.cc:492] T 8282e22bfb8b4d4994fa8b95ef154a68 P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:53.416110 6315 raft_consensus.cc:513] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.425634 5410 tablet_bootstrap.cc:654] T 8282e22bfb8b4d4994fa8b95ef154a68 P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:53.428196 6315 leader_election.cc:290] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:53.428321 4386 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 1b794133c1a346318e31bffb4ba6effb (table default.Table_b8d28650a552496b9582e1623ea694ff [id=574f9c9a69d34e918b567ac099e081e5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1b794133c1a346318e31bffb4ba6effb consensus state has no leader
I20250626 19:57:53.429525 6351 raft_consensus.cc:2241] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:53.429922 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e6136b0d3104ae88c3f4ae88a41de84" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:53.430898 5366 raft_consensus.cc:383] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:53.431201 5366 raft_consensus.cc:738] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:53.431269 4815 raft_consensus.cc:2466] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:53.431828 5366 consensus_queue.cc:260] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.432723 4609 leader_election.cc:304] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:53.432852 6351 raft_consensus.cc:2270] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:53.433553 6315 raft_consensus.cc:2802] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:53.433908 6315 raft_consensus.cc:491] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:53.434249 6315 raft_consensus.cc:3058] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:53.438621 6351 ts_tablet_manager.cc:1905] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:53.440089 5366 ts_tablet_manager.cc:1428] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.028s user 0.003s sys 0.002s
I20250626 19:57:53.441161 5366 tablet_bootstrap.cc:492] T 9e4673214ad84f75a861c3af82a13d8f P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:53.451978 4973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e6136b0d3104ae88c3f4ae88a41de84" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:53.452615 4973 raft_consensus.cc:2466] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:53.459530 5366 tablet_bootstrap.cc:654] T 9e4673214ad84f75a861c3af82a13d8f P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:53.468961 5353 tablet_bootstrap.cc:492] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:53.469380 5353 ts_tablet_manager.cc:1397] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.251s user 0.024s sys 0.007s
I20250626 19:57:53.473194 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 3143181fae16411888d74f9152f3d5cc with delete_type TABLET_DATA_DELETED (replaced by tablet 08045a939eed4ec69b49d68b555bdf80) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:53.471583 5353 raft_consensus.cc:357] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:53.474782 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 3143181fae16411888d74f9152f3d5cc because the tablet was not found. No further retry: Not found: Tablet not found: 3143181fae16411888d74f9152f3d5cc
I20250626 19:57:53.474884 5353 raft_consensus.cc:383] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:53.475030 4911 leader_election.cc:304] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86
I20250626 19:57:53.476472 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34b2919e5f534f4295336a4384be1f64" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:53.477356 5924 raft_consensus.cc:2802] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:53.477665 5924 raft_consensus.cc:491] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:53.477917 5924 raft_consensus.cc:3058] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:53.475106 5353 raft_consensus.cc:738] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:53.480733 5353 consensus_queue.cc:260] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.483883 5924 raft_consensus.cc:513] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.487241 5924 leader_election.cc:290] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:53.482731 5353 ts_tablet_manager.cc:1428] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20250626 19:57:53.488945 5353 tablet_bootstrap.cc:492] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
W20250626 19:57:53.489542 4383 catalog_manager.cc:5001] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): alter failed for tablet c990e4eca5cf4a8e8295e9599cf5be29 (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]),no further retry: Not found: Tablet not found: c990e4eca5cf4a8e8295e9599cf5be29
I20250626 19:57:53.491287 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet e7c352224d0a46c6a922bf0a0d765f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:40742
W20250626 19:57:53.492717 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet e7c352224d0a46c6a922bf0a0d765f17 because the tablet was not found. No further retry: Not found: Tablet not found: e7c352224d0a46c6a922bf0a0d765f17
I20250626 19:57:53.493117 4979 raft_consensus.cc:3058] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:57:53.494273 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet fc526a05990446a3b3bc973e79511476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:40742
W20250626 19:57:53.495589 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet fc526a05990446a3b3bc973e79511476 because the tablet was not found. No further retry: Not found: Tablet not found: fc526a05990446a3b3bc973e79511476
I20250626 19:57:53.497489 6242 consensus_queue.cc:1035] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:53.500164 4979 raft_consensus.cc:1273] T e4aee96ad3454d65b974c867605c4cdc P 5f8fb6d535ea42359fe34478df9e00a6 [term 2 FOLLOWER]: Refusing update from remote peer 93f885c6802341c889a40e9c20fbfff1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250626 19:57:53.512054 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 7851e464fcd44ecb82404d6a991db2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:48 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:53.512733 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e6136b0d3104ae88c3f4ae88a41de84" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:53.513307 4815 raft_consensus.cc:3058] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:53.513679 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 7851e464fcd44ecb82404d6a991db2a2 because the tablet was not found. No further retry: Not found: Tablet not found: 7851e464fcd44ecb82404d6a991db2a2
I20250626 19:57:53.517894 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:40742
W20250626 19:57:53.519572 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet f807cf54bbb146ff989762c18410e6bb because the tablet was not found. No further retry: Not found: Tablet not found: f807cf54bbb146ff989762c18410e6bb
I20250626 19:57:53.519531 4815 raft_consensus.cc:2466] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
W20250626 19:57:53.520295 4756 leader_election.cc:343] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:53.520669 4756 leader_election.cc:304] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:53.522337 6121 raft_consensus.cc:2747] T 34b2919e5f534f4295336a4384be1f64 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:53.524860 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 9172cb6576b240f6b25c3b1267618e6c with delete_type TABLET_DATA_DELETED (replaced by tablet 290232b0933b451c9d7744165bd72796) from {username='slave'} at 127.0.0.1:40742
W20250626 19:57:53.526345 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet 9172cb6576b240f6b25c3b1267618e6c because the tablet was not found. No further retry: Not found: Tablet not found: 9172cb6576b240f6b25c3b1267618e6c
I20250626 19:57:53.529249 6239 raft_consensus.cc:491] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:53.532707 6121 consensus_queue.cc:1035] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:53.534216 4911 leader_election.cc:304] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:53.535838 6241 raft_consensus.cc:2802] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:53.536298 6241 raft_consensus.cc:695] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
I20250626 19:57:53.535751 6239 raft_consensus.cc:513] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.537057 6241 consensus_queue.cc:237] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.540695 6239 leader_election.cc:290] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:53.542306 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 1feb6785d0c7459fbc6898b0c9d3e38c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:53.543038 6353 tablet_replica.cc:331] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:53.544008 6353 raft_consensus.cc:2241] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:53.544981 6353 raft_consensus.cc:2270] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:53.547240 6353 ts_tablet_manager.cc:1905] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:53.561219 5353 tablet_bootstrap.cc:654] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:53.563611 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet c990e4eca5cf4a8e8295e9599cf5be29 with delete_type TABLET_DATA_DELETED (replaced by tablet c2ff9841bbbd429b83c123e1ebdabfea) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:53.566530 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b794133c1a346318e31bffb4ba6effb" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:53.567142 4816 raft_consensus.cc:2466] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
W20250626 19:57:53.567458 4392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 80) took 6500 ms (client timeout 1000 ms). Trace:
I20250626 19:57:53.569731 4911 leader_election.cc:304] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:53.570767 4391 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 194) took 4576 ms. Trace:
I20250626 19:57:53.570868 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 54e68eba747e4a2a947e84e53a444d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:53.570916 6239 raft_consensus.cc:2802] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:53.571580 6239 raft_consensus.cc:491] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:53.572160 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f319c748374b7cbbf037ef55aecd1b with delete_type TABLET_DATA_DELETED (replaced by tablet 15aade4064d742488ce29145185551a7) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:53.571084 4391 rpcz_store.cc:276] 0626 19:57:48.994213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:52.950966 (+3956753us) service_pool.cc:225] Handling call
0626 19:57:52.972727 (+ 21761us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:53.285500 (+312773us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:53.569767 (+284267us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:53.570121 (+ 354us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:53.570539 (+ 418us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:53.454802 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:53.455322 (+ 520us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:53.455354 (+ 32us) write_op.cc:435] Acquired schema lock
0626 19:57:53.455373 (+ 19us) tablet.cc:660] Decoding operations
0626 19:57:53.455723 (+ 350us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:53.455787 (+ 64us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:53.455812 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:53.472938 (+ 17126us) tablet.cc:699] Row locks acquired
0626 19:57:53.472978 (+ 40us) write_op.cc:260] PREPARE: finished
0626 19:57:53.473120 (+ 142us) write_op.cc:270] Start()
0626 19:57:53.473230 (+ 110us) write_op.cc:276] Timestamp: P: 1750967873473389 usec, L: 0
0626 19:57:53.473257 (+ 27us) op_driver.cc:348] REPLICATION: starting
0626 19:57:53.473868 (+ 611us) log.cc:844] Serialized 433 byte log entry
0626 19:57:53.521755 (+ 47887us) op_driver.cc:464] REPLICATION: finished
0626 19:57:53.546948 (+ 25193us) write_op.cc:301] APPLY: starting
0626 19:57:53.547033 (+ 85us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:53.547150 (+ 117us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:53.547284 (+ 134us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:53.568733 (+ 21449us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:53.568974 (+ 241us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:53.569048 (+ 74us) write_op.cc:312] APPLY: finished
0626 19:57:53.569282 (+ 234us) log.cc:844] Serialized 29 byte log entry
0626 19:57:53.569435 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:53.569504 (+ 69us) write_op.cc:454] Released schema lock
0626 19:57:53.569544 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:53.569830 (+ 286us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:53.569854 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":22278,"apply.run_cpu_time_us":2037,"apply.run_wall_time_us":25507,"mutex_wait_us":700,"num_ops":2,"prepare.queue_time_us":167433,"prepare.run_cpu_time_us":8904,"prepare.run_wall_time_us":49780,"raft.queue_time_us":1229,"raft.run_cpu_time_us":1644,"raft.run_wall_time_us":1643,"replication_time_us":48331,"spinlock_wait_cycles":146176,"thread_start_us":220828,"threads_started":3,"wal-append.queue_time_us":28861}]]}
I20250626 19:57:53.577100 4655 tablet_service.cc:1515] Processing DeleteTablet for tablet e94a73b50b564a97bbee0f5015bb8805 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e4673214ad84f75a861c3af82a13d8f) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:53.577759 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet 9172cb6576b240f6b25c3b1267618e6c with delete_type TABLET_DATA_DELETED (replaced by tablet 290232b0933b451c9d7744165bd72796) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:53.577993 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e6136b0d3104ae88c3f4ae88a41de84" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:53.578768 6239 raft_consensus.cc:3058] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:53.581043 4656 tablet_service.cc:1515] Processing DeleteTablet for tablet fc526a05990446a3b3bc973e79511476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:53.581581 4651 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:53.583428 4655 tablet_service.cc:1515] Processing DeleteTablet for tablet 3143181fae16411888d74f9152f3d5cc with delete_type TABLET_DATA_DELETED (replaced by tablet 08045a939eed4ec69b49d68b555bdf80) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:53.583958 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet c990e4eca5cf4a8e8295e9599cf5be29 with delete_type TABLET_DATA_DELETED (replaced by tablet c2ff9841bbbd429b83c123e1ebdabfea) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:53.588464 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet e7c352224d0a46c6a922bf0a0d765f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:53.592999 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet 609d36d46bb84017b2a3a7ad48bfccfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:53.593115 6398 raft_consensus.cc:491] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:53.593581 6398 raft_consensus.cc:513] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.595532 6398 leader_election.cc:290] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:53.601059 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet 1feb6785d0c7459fbc6898b0c9d3e38c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:53.601783 4655 tablet_service.cc:1515] Processing DeleteTablet for tablet e7c352224d0a46c6a922bf0a0d765f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:53.602767 4655 tablet_service.cc:1515] Processing DeleteTablet for tablet d8be634cb36d40dca3f9338dab93ad2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:49 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:53.608983 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:53.610359 6351 ts_tablet_manager.cc:1918] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:53.610428 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet d8be634cb36d40dca3f9338dab93ad2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:49 UTC) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:53.624558 5366 tablet_bootstrap.cc:492] T 9e4673214ad84f75a861c3af82a13d8f P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:53.624976 5366 ts_tablet_manager.cc:1397] T 9e4673214ad84f75a861c3af82a13d8f P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.184s user 0.011s sys 0.010s
I20250626 19:57:53.625663 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e953f25481a245dd8edd61cd2e6061d1" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:53.634605 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e953f25481a245dd8edd61cd2e6061d1" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:53.635227 4816 raft_consensus.cc:2466] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
W20250626 19:57:53.567791 4392 rpcz_store.cc:269] 0626 19:57:47.066530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:47.090831 (+ 24301us) service_pool.cc:225] Handling call
0626 19:57:53.567273 (+6476442us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:48.722973 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:48.723431 (+ 458us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:48.723459 (+ 28us) write_op.cc:435] Acquired schema lock
0626 19:57:48.723480 (+ 21us) tablet.cc:660] Decoding operations
0626 19:57:48.723786 (+ 306us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:48.723825 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:48.723858 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:48.724007 (+ 149us) tablet.cc:699] Row locks acquired
0626 19:57:48.724023 (+ 16us) write_op.cc:260] PREPARE: finished
0626 19:57:48.724140 (+ 117us) write_op.cc:270] Start()
0626 19:57:48.724238 (+ 98us) write_op.cc:276] Timestamp: P: 1750967868724158 usec, L: 0
0626 19:57:48.724264 (+ 26us) op_driver.cc:348] REPLICATION: starting
0626 19:57:48.807457 (+ 83193us) log.cc:844] Serialized 356 byte log entry
0626 19:57:48.809211 (+ 1754us) spinlock_profiling.cc:243] Waited 82.7 ms on lock 0x7b54000203d4. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78f72b481 00007fe78f7a2a18 00007fe78fc0f811 00007fe78fc0efe8 00007fe78fc13c81 00007fe78fc13c39 00007fe78fc13bc9 00007fe78fc13b91 00007fe78fc12dbd 000000000059bcd4 000000000059bb09
0626 19:57:48.835721 (+ 26510us) op_driver.cc:464] REPLICATION: finished
0626 19:57:48.908268 (+ 72547us) write_op.cc:301] APPLY: starting
0626 19:57:48.908327 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:48.908430 (+ 103us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:48.908449 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:48.908622 (+ 173us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:48.908756 (+ 134us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:48.908797 (+ 41us) write_op.cc:312] APPLY: finished
0626 19:57:48.908986 (+ 189us) log.cc:844] Serialized 23 byte log entry
0626 19:57:48.909696 (+ 710us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:48.909755 (+ 59us) write_op.cc:454] Released schema lock
0626 19:57:48.909799 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:48.910172 (+ 373us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:48.910196 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":79,"child_traces":[["op",{"apply.queue_time_us":70537,"apply.run_cpu_time_us":1274,"apply.run_wall_time_us":3714,"num_ops":1,"prepare.queue_time_us":116427,"prepare.run_cpu_time_us":4774,"prepare.run_wall_time_us":86431,"raft.queue_time_us":191,"raft.run_cpu_time_us":582,"raft.run_wall_time_us":581,"replication_time_us":111361,"spinlock_wait_cycles":190211840}]]}
I20250626 19:57:53.643831 4392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_9df710adbb4946ff896ea61ff324f77f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:53.645627 6121 raft_consensus.cc:491] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 19:57:53.646224 4392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9df710adbb4946ff896ea61ff324f77f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:53.646030 6121 raft_consensus.cc:513] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.647630 6121 leader_election.cc:290] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:53.652365 5353 tablet_bootstrap.cc:492] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:53.654282 6121 raft_consensus.cc:491] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:53.655051 5353 ts_tablet_manager.cc:1397] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.166s user 0.018s sys 0.006s
I20250626 19:57:53.654678 6121 raft_consensus.cc:513] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:53.656409 4399 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:42351 (request call id 81) took 5039 ms (client timeout 1000 ms). Trace:
I20250626 19:57:53.657867 6121 leader_election.cc:290] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:53.658071 5353 raft_consensus.cc:357] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:53.656769 4399 rpcz_store.cc:269] 0626 19:57:48.616725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:48.788275 (+171550us) service_pool.cc:225] Handling call
0626 19:57:53.656224 (+4867949us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:51.179985 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:51.180550 (+ 565us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:51.180576 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:57:51.180592 (+ 16us) tablet.cc:660] Decoding operations
0626 19:57:51.180870 (+ 278us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:51.180936 (+ 66us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:51.180959 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:51.181114 (+ 155us) tablet.cc:699] Row locks acquired
0626 19:57:51.181129 (+ 15us) write_op.cc:260] PREPARE: finished
0626 19:57:51.181259 (+ 130us) write_op.cc:270] Start()
0626 19:57:51.181351 (+ 92us) write_op.cc:276] Timestamp: P: 1750967871181235 usec, L: 0
0626 19:57:51.181371 (+ 20us) op_driver.cc:348] REPLICATION: starting
0626 19:57:51.181886 (+ 515us) log.cc:844] Serialized 530 byte log entry
0626 19:57:51.410534 (+228648us) op_driver.cc:464] REPLICATION: finished
0626 19:57:51.451623 (+ 41089us) write_op.cc:301] APPLY: starting
0626 19:57:51.451680 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:51.451763 (+ 83us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:51.451871 (+ 108us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:51.452039 (+ 168us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:51.452177 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:51.452216 (+ 39us) write_op.cc:312] APPLY: finished
0626 19:57:51.452485 (+ 269us) log.cc:844] Serialized 23 byte log entry
0626 19:57:51.453230 (+ 745us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:51.453287 (+ 57us) write_op.cc:454] Released schema lock
0626 19:57:51.453317 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:51.453575 (+ 258us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:51.453600 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":312737,"reported_tablets":88,"child_traces":[["op",{"apply.queue_time_us":40914,"apply.run_cpu_time_us":1318,"apply.run_wall_time_us":1933,"num_ops":1,"prepare.queue_time_us":6170,"prepare.run_cpu_time_us":3283,"prepare.run_wall_time_us":73317,"raft.queue_time_us":198,"raft.run_cpu_time_us":1648,"raft.run_wall_time_us":1647,"replication_time_us":229053,"spinlock_wait_cycles":149248,"thread_start_us":82080,"threads_started":3,"wal-append.queue_time_us":30178,"wal-append.run_cpu_time_us":5213,"wal-append.run_wall_time_us":2224321}]]}
I20250626 19:57:53.662398 5924 raft_consensus.cc:491] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:53.662910 5924 raft_consensus.cc:513] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.663844 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet e7c352224d0a46c6a922bf0a0d765f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:53.663951 5353 raft_consensus.cc:383] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:53.664285 5353 raft_consensus.cc:738] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:53.664799 5924 leader_election.cc:290] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:53.666113 5353 consensus_queue.cc:260] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.669744 4957 tablet_service.cc:1515] Processing DeleteTablet for tablet 7851e464fcd44ecb82404d6a991db2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:48 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:53.670940 4958 tablet_service.cc:1515] Processing DeleteTablet for tablet c990e4eca5cf4a8e8295e9599cf5be29 with delete_type TABLET_DATA_DELETED (replaced by tablet c2ff9841bbbd429b83c123e1ebdabfea) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:53.672283 5353 ts_tablet_manager.cc:1428] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.016s user 0.002s sys 0.002s
I20250626 19:57:53.672583 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet e94a73b50b564a97bbee0f5015bb8805 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e4673214ad84f75a861c3af82a13d8f) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:53.673120 5353 tablet_bootstrap.cc:492] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:53.676717 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet 9172cb6576b240f6b25c3b1267618e6c with delete_type TABLET_DATA_DELETED (replaced by tablet 290232b0933b451c9d7744165bd72796) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:53.685158 5353 tablet_bootstrap.cc:654] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:53.613616 6318 raft_consensus.cc:491] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:53.688655 6318 raft_consensus.cc:513] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.623737 4655 tablet_service.cc:1515] Processing DeleteTablet for tablet 54e68eba747e4a2a947e84e53a444d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:53.632441 4673 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b794133c1a346318e31bffb4ba6effb" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:53.675063 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet e7c352224d0a46c6a922bf0a0d765f17 because the tablet was not found. No further retry: Not found: Tablet not found: e7c352224d0a46c6a922bf0a0d765f17
I20250626 19:57:53.642933 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f319c748374b7cbbf037ef55aecd1b with delete_type TABLET_DATA_DELETED (replaced by tablet 15aade4064d742488ce29145185551a7) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:53.649216 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "11648a3a9a94450284b8bc14ffe95852" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
W20250626 19:57:53.649849 4611 leader_election.cc:343] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:53.692643 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 7851e464fcd44ecb82404d6a991db2a2 because the tablet was not found. No further retry: Not found: Tablet not found: 7851e464fcd44ecb82404d6a991db2a2
W20250626 19:57:53.693146 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet c990e4eca5cf4a8e8295e9599cf5be29 because the tablet was not found. No further retry: Not found: Tablet not found: c990e4eca5cf4a8e8295e9599cf5be29
W20250626 19:57:53.693598 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet e94a73b50b564a97bbee0f5015bb8805 because the tablet was not found. No further retry: Not found: Tablet not found: e94a73b50b564a97bbee0f5015bb8805
W20250626 19:57:53.693787 4755 leader_election.cc:343] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:57:53.694033 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 9172cb6576b240f6b25c3b1267618e6c because the tablet was not found. No further retry: Not found: Tablet not found: 9172cb6576b240f6b25c3b1267618e6c
I20250626 19:57:53.690655 6318 leader_election.cc:290] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:53.696346 6351 log.cc:1199] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/7851e464fcd44ecb82404d6a991db2a2
I20250626 19:57:53.697039 6351 ts_tablet_manager.cc:1939] T 7851e464fcd44ecb82404d6a991db2a2 P 4d3be519a0044a2f921a503412f72e86: Deleting consensus metadata
I20250626 19:57:53.697551 4726 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:53.653472 6315 raft_consensus.cc:513] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.658017 4671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7def0bb1dbdc494eaa71748ebe1f8847" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:53.699424 4676 raft_consensus.cc:2391] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
W20250626 19:57:53.699985 4755 leader_election.cc:343] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:53.699772 6239 raft_consensus.cc:513] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.702945 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet 3143181fae16411888d74f9152f3d5cc with delete_type TABLET_DATA_DELETED (replaced by tablet 08045a939eed4ec69b49d68b555bdf80) from {username='slave'} at 127.0.0.1:41356
W20250626 19:57:53.704969 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet 3143181fae16411888d74f9152f3d5cc because the tablet was not found. No further retry: Not found: Tablet not found: 3143181fae16411888d74f9152f3d5cc
I20250626 19:57:53.705833 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet fc526a05990446a3b3bc973e79511476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:39 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:53.706542 6352 tablet_replica.cc:331] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:53.707389 6352 raft_consensus.cc:2241] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:53.707859 6352 raft_consensus.cc:2270] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:53.709826 6352 ts_tablet_manager.cc:1905] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:53.711160 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet e7c352224d0a46c6a922bf0a0d765f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:53.713891 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet e94a73b50b564a97bbee0f5015bb8805 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e4673214ad84f75a861c3af82a13d8f) from {username='slave'} at 127.0.0.1:41356
W20250626 19:57:53.679725 4397 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:52379 (request call id 86) took 6001 ms (client timeout 1000 ms). Trace:
W20250626 19:57:53.719749 4397 rpcz_store.cc:269] 0626 19:57:47.678080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:48.071776 (+393696us) service_pool.cc:225] Handling call
0626 19:57:53.679547 (+5607771us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":484078,"reported_tablets":84}
I20250626 19:57:53.669338 5410 tablet_bootstrap.cc:492] T 8282e22bfb8b4d4994fa8b95ef154a68 P 4d3be519a0044a2f921a503412f72e86: No bootstrap required, opened a new log
I20250626 19:57:53.721006 5410 ts_tablet_manager.cc:1397] T 8282e22bfb8b4d4994fa8b95ef154a68 P 4d3be519a0044a2f921a503412f72e86: Time spent bootstrapping tablet: real 0.303s user 0.018s sys 0.007s
I20250626 19:57:53.723193 5410 raft_consensus.cc:357] T 8282e22bfb8b4d4994fa8b95ef154a68 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:53.723814 5410 raft_consensus.cc:383] T 8282e22bfb8b4d4994fa8b95ef154a68 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:53.724058 5410 raft_consensus.cc:738] T 8282e22bfb8b4d4994fa8b95ef154a68 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Initialized, Role: FOLLOWER
I20250626 19:57:53.726894 6239 leader_election.cc:290] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:53.727922 6315 leader_election.cc:290] T 0e6136b0d3104ae88c3f4ae88a41de84 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:53.733204 4397 master_service.cc:948] ReplaceTablet: received request to replace tablet 419c4018256546ef9431ac2db619ee33 from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:53.734475 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54e68eba747e4a2a947e84e53a444d08" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:53.735018 4821 raft_consensus.cc:2466] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:53.735826 4821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b794133c1a346318e31bffb4ba6effb" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:53.736277 4821 raft_consensus.cc:3058] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:53.736847 4911 leader_election.cc:304] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:53.737715 6239 raft_consensus.cc:2802] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:53.738258 6239 raft_consensus.cc:491] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:53.738788 6239 raft_consensus.cc:3058] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:53.740113 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet f807cf54bbb146ff989762c18410e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:40 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:53.741314 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet 1feb6785d0c7459fbc6898b0c9d3e38c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:44 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:53.742450 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet d8be634cb36d40dca3f9338dab93ad2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:49 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:53.743014 4821 raft_consensus.cc:2466] T 1b794133c1a346318e31bffb4ba6effb P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 1.
I20250626 19:57:53.745052 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f319c748374b7cbbf037ef55aecd1b with delete_type TABLET_DATA_DELETED (replaced by tablet 15aade4064d742488ce29145185551a7) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:53.746166 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet 54e68eba747e4a2a947e84e53a444d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:46 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:53.751579 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet 7851e464fcd44ecb82404d6a991db2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:57:48 UTC) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:53.751766 4609 leader_election.cc:304] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters: 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:53.752882 6398 raft_consensus.cc:2802] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:53.753206 6398 raft_consensus.cc:491] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:53.753515 6398 raft_consensus.cc:3058] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:53.754205 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet c990e4eca5cf4a8e8295e9599cf5be29 with delete_type TABLET_DATA_DELETED (replaced by tablet c2ff9841bbbd429b83c123e1ebdabfea) from {username='slave'} at 127.0.0.1:41356
W20250626 19:57:53.738704 4398 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 71) took 6453 ms (client timeout 1000 ms). Trace:
I20250626 19:57:53.744287 4399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_96b230c0dd5f480ab71ae48a3c6dffe2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:57:53.760345 4399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96b230c0dd5f480ab71ae48a3c6dffe2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:53.761956 6398 raft_consensus.cc:513] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.763603 6353 ts_tablet_manager.cc:1918] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:53.764050 6353 log.cc:1199] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/1feb6785d0c7459fbc6898b0c9d3e38c
I20250626 19:57:53.764914 6353 ts_tablet_manager.cc:1939] T 1feb6785d0c7459fbc6898b0c9d3e38c P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
W20250626 19:57:53.766086 4390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:45585 (request call id 72) took 5227 ms (client timeout 1000 ms). Trace:
I20250626 19:57:53.767647 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet 3143181fae16411888d74f9152f3d5cc with delete_type TABLET_DATA_DELETED (replaced by tablet 08045a939eed4ec69b49d68b555bdf80) from {username='slave'} at 127.0.0.1:41356
W20250626 19:57:53.756714 4398 rpcz_store.cc:269] 0626 19:57:47.284743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:47.970281 (+685538us) service_pool.cc:225] Handling call
0626 19:57:53.738528 (+5768247us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":832403,"reported_tablets":81}
I20250626 19:57:53.771783 4398 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34796:
name: "default.Table_26f37cbdbed34b6b9528b2610ec0529f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:57:53.772003 5366 raft_consensus.cc:357] T 9e4673214ad84f75a861c3af82a13d8f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.772660 5366 raft_consensus.cc:383] T 9e4673214ad84f75a861c3af82a13d8f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:53.772888 5366 raft_consensus.cc:738] T 9e4673214ad84f75a861c3af82a13d8f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
W20250626 19:57:53.773422 4398 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26f37cbdbed34b6b9528b2610ec0529f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:57:53.773402 5366 consensus_queue.cc:260] T 9e4673214ad84f75a861c3af82a13d8f P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.775202 5366 ts_tablet_manager.cc:1428] T 9e4673214ad84f75a861c3af82a13d8f P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.150s user 0.007s sys 0.000s
I20250626 19:57:53.776168 4392 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 195) took 4769 ms. Trace:
I20250626 19:57:53.779400 5366 tablet_bootstrap.cc:492] T 2999736d3ea4428b8ce9668aeef52823 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:53.786412 6398 leader_election.cc:290] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:53.792008 4609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 124434us
I20250626 19:57:53.794415 5366 tablet_bootstrap.cc:654] T 2999736d3ea4428b8ce9668aeef52823 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:53.800534 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54e68eba747e4a2a947e84e53a444d08" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:53.801199 4815 raft_consensus.cc:2466] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 0.
I20250626 19:57:53.805387 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e6136b0d3104ae88c3f4ae88a41de84" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:53.806098 4815 raft_consensus.cc:2391] T 0e6136b0d3104ae88c3f4ae88a41de84 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:53.808531 6239 raft_consensus.cc:513] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.810578 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e953f25481a245dd8edd61cd2e6061d1" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:53.811141 4815 raft_consensus.cc:3058] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:53.815898 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54e68eba747e4a2a947e84e53a444d08" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:53.816459 4676 raft_consensus.cc:2466] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:53.831032 5410 consensus_queue.cc:260] T 8282e22bfb8b4d4994fa8b95ef154a68 P 4d3be519a0044a2f921a503412f72e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:53.831852 6399 raft_consensus.cc:491] T 8282e22bfb8b4d4994fa8b95ef154a68 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:53.832266 6399 raft_consensus.cc:513] T 8282e22bfb8b4d4994fa8b95ef154a68 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
W20250626 19:57:53.766417 4390 rpcz_store.cc:269] 0626 19:57:48.538371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:48.592915 (+ 54544us) service_pool.cc:225] Handling call
0626 19:57:53.765924 (+5173009us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:52.152909 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:52.153456 (+ 547us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:52.153494 (+ 38us) write_op.cc:435] Acquired schema lock
0626 19:57:52.153514 (+ 20us) tablet.cc:660] Decoding operations
0626 19:57:52.153848 (+ 334us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:52.153910 (+ 62us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:52.153934 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:57:52.154097 (+ 163us) tablet.cc:699] Row locks acquired
0626 19:57:52.154111 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:57:52.154239 (+ 128us) write_op.cc:270] Start()
0626 19:57:52.154334 (+ 95us) write_op.cc:276] Timestamp: P: 1750967872154195 usec, L: 0
0626 19:57:52.154354 (+ 20us) op_driver.cc:348] REPLICATION: starting
0626 19:57:52.169169 (+ 14815us) log.cc:844] Serialized 530 byte log entry
0626 19:57:52.265166 (+ 95997us) op_driver.cc:464] REPLICATION: finished
0626 19:57:52.272713 (+ 7547us) write_op.cc:301] APPLY: starting
0626 19:57:52.272771 (+ 58us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:52.272855 (+ 84us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:52.272957 (+ 102us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:52.273114 (+ 157us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:52.273243 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:52.273287 (+ 44us) write_op.cc:312] APPLY: finished
0626 19:57:52.273457 (+ 170us) log.cc:844] Serialized 23 byte log entry
0626 19:57:52.304246 (+ 30789us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:52.304330 (+ 84us) write_op.cc:454] Released schema lock
0626 19:57:52.304393 (+ 63us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:52.304735 (+ 342us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:52.304770 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":39646,"reported_tablets":93,"child_traces":[["op",{"apply.queue_time_us":7355,"apply.run_cpu_time_us":1417,"apply.run_wall_time_us":31973,"num_ops":1,"prepare.queue_time_us":56098,"prepare.run_cpu_time_us":5682,"prepare.run_wall_time_us":110905,"raft.queue_time_us":49887,"raft.run_cpu_time_us":1644,"raft.run_wall_time_us":40951,"replication_time_us":110666,"thread_start_us":104087,"threads_started":3}]]}
I20250626 19:57:53.834491 6399 leader_election.cc:290] T 8282e22bfb8b4d4994fa8b95ef154a68 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951), 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711)
I20250626 19:57:53.836458 5410 ts_tablet_manager.cc:1428] T 8282e22bfb8b4d4994fa8b95ef154a68 P 4d3be519a0044a2f921a503412f72e86: Time spent starting tablet: real 0.115s user 0.004s sys 0.000s
I20250626 19:57:53.836560 4390 master_service.cc:948] ReplaceTablet: received request to replace tablet f70c620bf4474e59aa1f86761aca0daa from {username='slave'} at 127.0.0.1:34796
I20250626 19:57:53.838091 4609 leader_election.cc:304] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:53.839138 6399 raft_consensus.cc:2802] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:53.839470 6399 raft_consensus.cc:491] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:53.839766 6399 raft_consensus.cc:3058] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:53.820365 5353 tablet_bootstrap.cc:492] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:53.841689 5353 ts_tablet_manager.cc:1397] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.169s user 0.011s sys 0.001s
I20250626 19:57:53.843967 5353 raft_consensus.cc:357] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:53.844647 5353 raft_consensus.cc:383] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:53.844894 5353 raft_consensus.cc:738] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:53.845468 5353 consensus_queue.cc:260] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:53.847347 6239 leader_election.cc:290] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:53.847703 5353 ts_tablet_manager.cc:1428] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250626 19:57:53.849735 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8282e22bfb8b4d4994fa8b95ef154a68" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:53.850430 4816 raft_consensus.cc:2391] T 8282e22bfb8b4d4994fa8b95ef154a68 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:53.853614 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 1feb6785d0c7459fbc6898b0c9d3e38c (table default.Table_987278d268e04477828ad87c6d9c8c01 [id=da221881c13940a4b0dfeb88aa81c932]) successfully deleted
W20250626 19:57:53.854385 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet c990e4eca5cf4a8e8295e9599cf5be29 because the tablet was not found. No further retry: Not found: Tablet not found: c990e4eca5cf4a8e8295e9599cf5be29
I20250626 19:57:53.854619 6353 tablet_replica.cc:331] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:53.855659 6353 raft_consensus.cc:2241] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:53.856138 5353 tablet_bootstrap.cc:492] T 11648a3a9a94450284b8bc14ffe95852 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:53.865299 4384 catalog_manager.cc:4928] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): tablet 7851e464fcd44ecb82404d6a991db2a2 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd]) successfully deleted
I20250626 19:57:53.876567 5410 tablet_bootstrap.cc:492] T 1b794133c1a346318e31bffb4ba6effb P 4d3be519a0044a2f921a503412f72e86: Bootstrap starting.
I20250626 19:57:53.878988 5353 tablet_bootstrap.cc:654] T 11648a3a9a94450284b8bc14ffe95852 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:53.879832 6351 tablet_replica.cc:331] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86: stopping tablet replica
I20250626 19:57:53.879964 4676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b794133c1a346318e31bffb4ba6effb" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:53.881163 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54e68eba747e4a2a947e84e53a444d08" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86"
I20250626 19:57:53.885888 6399 raft_consensus.cc:513] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.886986 4669 raft_consensus.cc:2391] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:53.776506 4392 rpcz_store.cc:276] 0626 19:57:49.006825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:53.643316 (+4636491us) service_pool.cc:225] Handling call
0626 19:57:53.689384 (+ 46068us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:53.690860 (+ 1476us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:53.775567 (+ 84707us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:53.775865 (+ 298us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:53.776104 (+ 239us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:53.692275 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:53.692751 (+ 476us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:53.692775 (+ 24us) write_op.cc:435] Acquired schema lock
0626 19:57:53.692787 (+ 12us) tablet.cc:660] Decoding operations
0626 19:57:53.693112 (+ 325us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:53.693161 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:53.693183 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:53.693444 (+ 261us) tablet.cc:699] Row locks acquired
0626 19:57:53.693459 (+ 15us) write_op.cc:260] PREPARE: finished
0626 19:57:53.693576 (+ 117us) write_op.cc:270] Start()
0626 19:57:53.693666 (+ 90us) write_op.cc:276] Timestamp: P: 1750967873693938 usec, L: 0
0626 19:57:53.693684 (+ 18us) op_driver.cc:348] REPLICATION: starting
0626 19:57:53.694172 (+ 488us) log.cc:844] Serialized 433 byte log entry
0626 19:57:53.704239 (+ 10067us) op_driver.cc:464] REPLICATION: finished
0626 19:57:53.704577 (+ 338us) write_op.cc:301] APPLY: starting
0626 19:57:53.704648 (+ 71us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:53.704753 (+ 105us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:53.704836 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:53.705241 (+ 405us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:53.705372 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:53.705417 (+ 45us) write_op.cc:312] APPLY: finished
0626 19:57:53.705580 (+ 163us) log.cc:844] Serialized 29 byte log entry
0626 19:57:53.706231 (+ 651us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:53.706305 (+ 74us) write_op.cc:454] Released schema lock
0626 19:57:53.706339 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:53.706565 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:53.706589 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":186,"apply.run_cpu_time_us":1453,"apply.run_wall_time_us":1949,"num_ops":2,"prepare.queue_time_us":100,"prepare.run_cpu_time_us":2354,"prepare.run_wall_time_us":2357,"raft.queue_time_us":1140,"raft.run_cpu_time_us":1214,"raft.run_wall_time_us":1213,"replication_time_us":10461}]]}
I20250626 19:57:53.901008 4392 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:56494 (request call id 1) took 3597 ms. Trace:
I20250626 19:57:53.901332 4392 rpcz_store.cc:276] 0626 19:57:50.303709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:53.900453 (+3596744us) service_pool.cc:225] Handling call
0626 19:57:53.900932 (+ 479us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 19:57:53.921837 4912 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 55753us
I20250626 19:57:53.931870 4397 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet 419c4018256546ef9431ac2db619ee33
I20250626 19:57:53.937047 4911 leader_election.cc:304] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:53.938341 6242 raft_consensus.cc:2802] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:53.938889 6242 raft_consensus.cc:695] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Running, Role: LEADER
W20250626 19:57:53.938345 4912 leader_election.cc:343] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:53.941320 6242 consensus_queue.cc:237] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.902726 4869 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:53.947072 6353 raft_consensus.cc:2270] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:53.947357 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet 419c4018256546ef9431ac2db619ee33 with delete_type TABLET_DATA_DELETED (replaced by tablet d31211102ae04c2a8f71e3b9ec9881c1) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:53.956063 6121 raft_consensus.cc:491] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:53.956538 6121 raft_consensus.cc:513] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.958263 6121 leader_election.cc:290] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:53.959275 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet 419c4018256546ef9431ac2db619ee33 with delete_type TABLET_DATA_DELETED (replaced by tablet d31211102ae04c2a8f71e3b9ec9881c1) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:53.959609 4669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "134c7a645f5f4eb89b87929fb67dac97" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3be519a0044a2f921a503412f72e86" is_pre_election: true
I20250626 19:57:53.959910 4397 catalog_manager.cc:6652] ReplaceTablet: tablet 419c4018256546ef9431ac2db619ee33 deleted and replaced by tablet d31211102ae04c2a8f71e3b9ec9881c1
I20250626 19:57:53.960258 4397 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34796 (request call id 196) took 4005 ms. Trace:
W20250626 19:57:53.960975 4755 leader_election.cc:343] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:53.962939 5410 tablet_bootstrap.cc:654] T 1b794133c1a346318e31bffb4ba6effb P 4d3be519a0044a2f921a503412f72e86: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:53.970655 4398 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 198) took 3994 ms. Trace:
I20250626 19:57:53.971038 4398 rpcz_store.cc:276] 0626 19:57:49.976411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:53.771249 (+3794838us) service_pool.cc:225] Handling call
0626 19:57:53.773767 (+ 2518us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:53.791548 (+ 17781us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:53.970074 (+178526us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:53.970348 (+ 274us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:53.970599 (+ 251us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:53.911148 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:53.911597 (+ 449us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:53.911624 (+ 27us) write_op.cc:435] Acquired schema lock
0626 19:57:53.911644 (+ 20us) tablet.cc:660] Decoding operations
0626 19:57:53.912126 (+ 482us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:53.912180 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:53.912218 (+ 38us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:53.912491 (+ 273us) tablet.cc:699] Row locks acquired
0626 19:57:53.912515 (+ 24us) write_op.cc:260] PREPARE: finished
0626 19:57:53.914236 (+ 1721us) write_op.cc:270] Start()
0626 19:57:53.914350 (+ 114us) write_op.cc:276] Timestamp: P: 1750967873914105 usec, L: 0
0626 19:57:53.914380 (+ 30us) op_driver.cc:348] REPLICATION: starting
0626 19:57:53.914973 (+ 593us) log.cc:844] Serialized 433 byte log entry
0626 19:57:53.965817 (+ 50844us) op_driver.cc:464] REPLICATION: finished
0626 19:57:53.968319 (+ 2502us) write_op.cc:301] APPLY: starting
0626 19:57:53.968367 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:53.968461 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:53.968543 (+ 82us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:53.968958 (+ 415us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:53.969108 (+ 150us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:53.969150 (+ 42us) write_op.cc:312] APPLY: finished
0626 19:57:53.969336 (+ 186us) log.cc:844] Serialized 29 byte log entry
0626 19:57:53.969462 (+ 126us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:53.969526 (+ 64us) write_op.cc:454] Released schema lock
0626 19:57:53.969561 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:53.970105 (+ 544us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:53.970127 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2372,"apply.run_cpu_time_us":1752,"apply.run_wall_time_us":1752,"mutex_wait_us":102479,"num_ops":2,"prepare.queue_time_us":118321,"prepare.run_cpu_time_us":3040,"prepare.run_wall_time_us":5094,"replication_time_us":51363}]]}
I20250626 19:57:53.975862 4399 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34796 (request call id 197) took 4002 ms. Trace:
W20250626 19:57:53.976497 4912 leader_election.cc:343] T 1b794133c1a346318e31bffb4ba6effb P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:53.979455 4977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54e68eba747e4a2a947e84e53a444d08" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:53.976202 4399 rpcz_store.cc:276] 0626 19:57:49.973725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:53.743875 (+3770150us) service_pool.cc:225] Handling call
0626 19:57:53.760814 (+ 16939us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:57:53.762306 (+ 1492us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:57:53.975279 (+212973us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:57:53.975570 (+ 291us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:57:53.975809 (+ 239us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:53.916418 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:53.916932 (+ 514us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:53.916967 (+ 35us) write_op.cc:435] Acquired schema lock
0626 19:57:53.942712 (+ 25745us) tablet.cc:660] Decoding operations
0626 19:57:53.943083 (+ 371us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:53.943125 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:53.943151 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:53.943376 (+ 225us) tablet.cc:699] Row locks acquired
0626 19:57:53.943393 (+ 17us) write_op.cc:260] PREPARE: finished
0626 19:57:53.943524 (+ 131us) write_op.cc:270] Start()
0626 19:57:53.944170 (+ 646us) write_op.cc:276] Timestamp: P: 1750967873943413 usec, L: 0
0626 19:57:53.944195 (+ 25us) op_driver.cc:348] REPLICATION: starting
0626 19:57:53.944725 (+ 530us) log.cc:844] Serialized 433 byte log entry
0626 19:57:53.973475 (+ 28750us) op_driver.cc:464] REPLICATION: finished
0626 19:57:53.973788 (+ 313us) write_op.cc:301] APPLY: starting
0626 19:57:53.973857 (+ 69us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:53.973958 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:53.974076 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:53.974475 (+ 399us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:53.974610 (+ 135us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:57:53.974650 (+ 40us) write_op.cc:312] APPLY: finished
0626 19:57:53.974849 (+ 199us) log.cc:844] Serialized 29 byte log entry
0626 19:57:53.974990 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:53.975060 (+ 70us) write_op.cc:454] Released schema lock
0626 19:57:53.975094 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":155,"apply.run_cpu_time_us":1526,"apply.run_wall_time_us":1525,"mutex_wait_us":33624,"num_ops":2,"prepare.queue_time_us":53748,"prepare.run_cpu_time_us":2919,"prepare.run_wall_time_us":61331,"replication_time_us":29184}]]}
I20250626 19:57:53.979941 4971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7def0bb1dbdc494eaa71748ebe1f8847" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:53.980170 4977 raft_consensus.cc:2391] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:53.980984 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "134c7a645f5f4eb89b87929fb67dac97" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:53.920272 4815 raft_consensus.cc:2466] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:53.981956 4970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e6136b0d3104ae88c3f4ae88a41de84" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:53.982481 4282 master-stress-test.cc:579] Killed test thread 1/10
I20250626 19:57:53.982722 4282 master-stress-test.cc:577] Killing test thread 2/10
I20250626 19:57:53.982996 4282 master-stress-test.cc:579] Killed test thread 2/10
I20250626 19:57:53.983192 4282 master-stress-test.cc:577] Killing test thread 3/10
I20250626 19:57:53.983418 4282 master-stress-test.cc:579] Killed test thread 3/10
I20250626 19:57:53.983618 4282 master-stress-test.cc:577] Killing test thread 4/10
I20250626 19:57:53.983548 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet 419c4018256546ef9431ac2db619ee33 with delete_type TABLET_DATA_DELETED (replaced by tablet d31211102ae04c2a8f71e3b9ec9881c1) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:53.983827 4282 master-stress-test.cc:579] Killed test thread 4/10
I20250626 19:57:53.984023 4282 master-stress-test.cc:577] Killing test thread 5/10
I20250626 19:57:53.984251 4282 master-stress-test.cc:579] Killed test thread 5/10
I20250626 19:57:53.984436 4282 master-stress-test.cc:577] Killing test thread 6/10
I20250626 19:57:53.984912 4282 master-stress-test.cc:579] Killed test thread 6/10
I20250626 19:57:53.984899 6352 ts_tablet_manager.cc:1918] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:53.985133 4282 master-stress-test.cc:577] Killing test thread 7/10
I20250626 19:57:53.985263 6352 log.cc:1199] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/fc526a05990446a3b3bc973e79511476
I20250626 19:57:53.985476 4282 master-stress-test.cc:579] Killed test thread 7/10
I20250626 19:57:53.930956 5366 tablet_bootstrap.cc:492] T 2999736d3ea4428b8ce9668aeef52823 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:53.934149 4816 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54e68eba747e4a2a947e84e53a444d08" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:53.985675 4282 master-stress-test.cc:577] Killing test thread 8/10
I20250626 19:57:53.985898 4282 master-stress-test.cc:579] Killed test thread 8/10
I20250626 19:57:53.986088 4282 master-stress-test.cc:577] Killing test thread 9/10
I20250626 19:57:53.986511 5366 ts_tablet_manager.cc:1397] T 2999736d3ea4428b8ce9668aeef52823 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.207s user 0.010s sys 0.000s
I20250626 19:57:53.988292 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "11648a3a9a94450284b8bc14ffe95852" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:53.988776 5366 raft_consensus.cc:357] T 2999736d3ea4428b8ce9668aeef52823 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.989974 5924 raft_consensus.cc:491] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:53.990360 5924 raft_consensus.cc:513] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.989374 5366 raft_consensus.cc:383] T 2999736d3ea4428b8ce9668aeef52823 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:53.991710 5366 raft_consensus.cc:738] T 2999736d3ea4428b8ce9668aeef52823 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:53.991886 5924 leader_election.cc:290] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:53.992360 5366 consensus_queue.cc:260] T 2999736d3ea4428b8ce9668aeef52823 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.993484 5924 consensus_queue.cc:1035] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:57:53.993999 6351 raft_consensus.cc:2241] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:53.994550 6351 raft_consensus.cc:2270] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:53.996481 6351 ts_tablet_manager.cc:1905] T e94a73b50b564a97bbee0f5015bb8805 P 4d3be519a0044a2f921a503412f72e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:53.999783 4609 leader_election.cc:304] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:54.000481 6315 raft_consensus.cc:2802] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:57:54.000907 6315 raft_consensus.cc:695] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3be519a0044a2f921a503412f72e86, State: Running, Role: LEADER
I20250626 19:57:54.001634 6315 consensus_queue.cc:237] T e953f25481a245dd8edd61cd2e6061d1 P 4d3be519a0044a2f921a503412f72e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:53.960528 4397 rpcz_store.cc:276] 0626 19:57:49.955085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:53.732957 (+3777872us) service_pool.cc:225] Handling call
0626 19:57:53.747846 (+ 14889us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5400002490. stack: 00007fe788829483 00007fe788829380 00007fe7883a5574 00000000004fa174 00000000004fa109 00007fe78869e9a1 00007fe78881deb1 00007fe78881e3f2 00007fe7914f24f9 00007fe79148b996 00007fe79145ce68 00007fe791559891 00007fe78f2943a7 00007fe78f294332 00007fe78f294261 00007fe78f2941dc
0626 19:57:53.960182 (+212336us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:53.892154 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:53.892672 (+ 518us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:53.892712 (+ 40us) write_op.cc:435] Acquired schema lock
0626 19:57:53.892751 (+ 39us) tablet.cc:660] Decoding operations
0626 19:57:53.893207 (+ 456us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:53.893265 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:53.893306 (+ 41us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:53.893611 (+ 305us) tablet.cc:699] Row locks acquired
0626 19:57:53.893636 (+ 25us) write_op.cc:260] PREPARE: finished
0626 19:57:53.893802 (+ 166us) write_op.cc:270] Start()
0626 19:57:53.893919 (+ 117us) write_op.cc:276] Timestamp: P: 1750967873893661 usec, L: 0
0626 19:57:53.893947 (+ 28us) op_driver.cc:348] REPLICATION: starting
0626 19:57:53.894691 (+ 744us) log.cc:844] Serialized 659 byte log entry
0626 19:57:53.921409 (+ 26718us) op_driver.cc:464] REPLICATION: finished
0626 19:57:53.921747 (+ 338us) write_op.cc:301] APPLY: starting
0626 19:57:53.921798 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:53.921908 (+ 110us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:53.921998 (+ 90us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:53.922376 (+ 378us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:53.922628 (+ 252us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:53.922694 (+ 66us) write_op.cc:312] APPLY: finished
0626 19:57:53.923154 (+ 460us) log.cc:844] Serialized 29 byte log entry
0626 19:57:53.923295 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:53.923360 (+ 65us) write_op.cc:454] Released schema lock
0626 19:57:53.923389 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0626 19:57:53.923605 (+ 216us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:53.923626 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":24597760,"child_traces":[["op",{"apply.queue_time_us":193,"apply.run_cpu_time_us":1595,"apply.run_wall_time_us":1825,"num_ops":2,"prepare.queue_time_us":140761,"prepare.run_cpu_time_us":3051,"prepare.run_wall_time_us":11569,"raft.queue_time_us":6760,"raft.run_cpu_time_us":1644,"raft.run_wall_time_us":9221,"replication_time_us":27361}]]}
I20250626 19:57:54.014587 4282 master-stress-test.cc:579] Killed test thread 9/10
I20250626 19:57:54.014895 4282 master-stress-test.cc:577] Killing test thread 10/10
W20250626 19:57:54.015339 4911 leader_election.cc:343] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:54.015709 4911 leader_election.cc:304] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:54.016041 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e953f25481a245dd8edd61cd2e6061d1" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:54.016592 6239 raft_consensus.cc:2747] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:57:54.016733 4815 raft_consensus.cc:2391] T e953f25481a245dd8edd61cd2e6061d1 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f8fb6d535ea42359fe34478df9e00a6 in current term 1: Already voted for candidate 4d3be519a0044a2f921a503412f72e86 in this term.
I20250626 19:57:53.981439 6353 ts_tablet_manager.cc:1905] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:54.028442 4390 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 62ac80df82774b9b87c0be076b5637af: Sending DeleteTablet for 3 replicas of tablet f70c620bf4474e59aa1f86761aca0daa
I20250626 19:57:54.029695 4390 catalog_manager.cc:6652] ReplaceTablet: tablet f70c620bf4474e59aa1f86761aca0daa deleted and replaced by tablet 4512528e00b14a0c8ad23a721fc5b103
I20250626 19:57:54.029594 6399 leader_election.cc:290] T 54e68eba747e4a2a947e84e53a444d08 P 4d3be519a0044a2f921a503412f72e86 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:54.029994 4390 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34796 (request call id 199) took 3869 ms. Trace:
I20250626 19:57:54.032051 4795 tablet_service.cc:1515] Processing DeleteTablet for tablet f70c620bf4474e59aa1f86761aca0daa with delete_type TABLET_DATA_DELETED (replaced by tablet 4512528e00b14a0c8ad23a721fc5b103) from {username='slave'} at 127.0.0.1:40742
I20250626 19:57:54.030241 4390 rpcz_store.cc:276] 0626 19:57:50.160740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:53.836428 (+3675688us) service_pool.cc:225] Handling call
0626 19:57:54.029936 (+193508us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:57:53.978218 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:57:53.978931 (+ 713us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:57:53.978967 (+ 36us) write_op.cc:435] Acquired schema lock
0626 19:57:53.978987 (+ 20us) tablet.cc:660] Decoding operations
0626 19:57:54.016134 (+ 37147us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:57:54.016180 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0626 19:57:54.016207 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:57:54.016445 (+ 238us) tablet.cc:699] Row locks acquired
0626 19:57:54.016462 (+ 17us) write_op.cc:260] PREPARE: finished
0626 19:57:54.016583 (+ 121us) write_op.cc:270] Start()
0626 19:57:54.016678 (+ 95us) write_op.cc:276] Timestamp: P: 1750967874016503 usec, L: 0
0626 19:57:54.016697 (+ 19us) op_driver.cc:348] REPLICATION: starting
0626 19:57:54.017192 (+ 495us) log.cc:844] Serialized 625 byte log entry
0626 19:57:54.025633 (+ 8441us) op_driver.cc:464] REPLICATION: finished
0626 19:57:54.025992 (+ 359us) write_op.cc:301] APPLY: starting
0626 19:57:54.026040 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
0626 19:57:54.026138 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
0626 19:57:54.026229 (+ 91us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:57:54.027151 (+ 922us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:57:54.027386 (+ 235us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:57:54.027449 (+ 63us) write_op.cc:312] APPLY: finished
0626 19:57:54.027629 (+ 180us) log.cc:844] Serialized 29 byte log entry
0626 19:57:54.027792 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:57:54.027858 (+ 66us) write_op.cc:454] Released schema lock
0626 19:57:54.027897 (+ 39us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":209,"apply.run_cpu_time_us":2108,"apply.run_wall_time_us":2104,"mutex_wait_us":30141,"num_ops":2,"prepare.queue_time_us":100926,"raft.queue_time_us":102,"raft.run_cpu_time_us":786,"raft.run_wall_time_us":786,"replication_time_us":8846}]]}
I20250626 19:57:54.033522 4657 tablet_service.cc:1515] Processing DeleteTablet for tablet f70c620bf4474e59aa1f86761aca0daa with delete_type TABLET_DATA_DELETED (replaced by tablet 4512528e00b14a0c8ad23a721fc5b103) from {username='slave'} at 127.0.0.1:49550
I20250626 19:57:54.038194 4282 master-stress-test.cc:579] Killed test thread 10/10
I20250626 19:57:54.038582 4282 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu with pid 4562
I20250626 19:57:54.043629 4972 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e953f25481a245dd8edd61cd2e6061d1" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:54.044201 4972 raft_consensus.cc:3058] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:57:54.044595 4959 tablet_service.cc:1515] Processing DeleteTablet for tablet f70c620bf4474e59aa1f86761aca0daa with delete_type TABLET_DATA_DELETED (replaced by tablet 4512528e00b14a0c8ad23a721fc5b103) from {username='slave'} at 127.0.0.1:41356
I20250626 19:57:54.046425 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54e68eba747e4a2a947e84e53a444d08" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1"
I20250626 19:57:54.049054 5353 tablet_bootstrap.cc:492] T 11648a3a9a94450284b8bc14ffe95852 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:54.049477 5353 ts_tablet_manager.cc:1397] T 11648a3a9a94450284b8bc14ffe95852 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.194s user 0.005s sys 0.007s
I20250626 19:57:54.051316 4972 raft_consensus.cc:2466] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3be519a0044a2f921a503412f72e86 in term 1.
I20250626 19:57:54.051648 5353 raft_consensus.cc:357] T 11648a3a9a94450284b8bc14ffe95852 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:54.052250 5353 raft_consensus.cc:383] T 11648a3a9a94450284b8bc14ffe95852 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:54.052474 5353 raft_consensus.cc:738] T 11648a3a9a94450284b8bc14ffe95852 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:54.054308 4979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8282e22bfb8b4d4994fa8b95ef154a68" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:54.056511 4975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54e68eba747e4a2a947e84e53a444d08" candidate_uuid: "4d3be519a0044a2f921a503412f72e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6"
I20250626 19:57:54.057215 4975 raft_consensus.cc:2391] T 54e68eba747e4a2a947e84e53a444d08 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d3be519a0044a2f921a503412f72e86 in current term 1: Already voted for candidate 5f8fb6d535ea42359fe34478df9e00a6 in this term.
I20250626 19:57:54.063222 5353 consensus_queue.cc:260] T 11648a3a9a94450284b8bc14ffe95852 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:54.072798 5366 ts_tablet_manager.cc:1428] T 2999736d3ea4428b8ce9668aeef52823 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.086s user 0.004s sys 0.000s
I20250626 19:57:54.073691 5366 tablet_bootstrap.cc:492] T f83ef2b2b82b4fa4996b12ce3d8b5401 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
W20250626 19:57:54.097688 4756 leader_election.cc:343] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:54.098150 4756 leader_election.cc:304] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:54.071928 6352 ts_tablet_manager.cc:1939] T fc526a05990446a3b3bc973e79511476 P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:54.102466 6121 raft_consensus.cc:2747] T 7def0bb1dbdc494eaa71748ebe1f8847 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:54.103536 4756 leader_election.cc:343] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:54.103921 4756 leader_election.cc:304] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:54.123208 5366 tablet_bootstrap.cc:654] T f83ef2b2b82b4fa4996b12ce3d8b5401 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:54.126870 6121 raft_consensus.cc:2747] T 134c7a645f5f4eb89b87929fb67dac97 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:54.131994 5353 ts_tablet_manager.cc:1428] T 11648a3a9a94450284b8bc14ffe95852 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.082s user 0.005s sys 0.001s
I20250626 19:57:54.132880 5353 tablet_bootstrap.cc:492] T 56f319c748374b7cbbf037ef55aecd1b P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:54.140786 5353 tablet_bootstrap.cc:654] T 56f319c748374b7cbbf037ef55aecd1b P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:54.142485 5366 tablet_bootstrap.cc:492] T f83ef2b2b82b4fa4996b12ce3d8b5401 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:54.147310 5366 ts_tablet_manager.cc:1397] T f83ef2b2b82b4fa4996b12ce3d8b5401 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.000s
I20250626 19:57:54.149828 5366 raft_consensus.cc:357] T f83ef2b2b82b4fa4996b12ce3d8b5401 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:54.150534 5366 raft_consensus.cc:383] T f83ef2b2b82b4fa4996b12ce3d8b5401 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:54.155171 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet fc526a05990446a3b3bc973e79511476 (table default.Table_38eec4fd44e848378efabe2ebbc88983 [id=4e600070f9e44758a4b11f0adeab28e5]) successfully deleted
I20250626 19:57:54.155761 6353 ts_tablet_manager.cc:1918] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:57:54.156170 6353 log.cc:1199] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/54e68eba747e4a2a947e84e53a444d08
I20250626 19:57:54.156827 6353 ts_tablet_manager.cc:1939] T 54e68eba747e4a2a947e84e53a444d08 P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
W20250626 19:57:54.157842 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet e7c352224d0a46c6a922bf0a0d765f17 because the tablet was not found. No further retry: Not found: Tablet not found: e7c352224d0a46c6a922bf0a0d765f17
W20250626 19:57:54.159394 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet e94a73b50b564a97bbee0f5015bb8805 because the tablet was not found. No further retry: Not found: Tablet not found: e94a73b50b564a97bbee0f5015bb8805
I20250626 19:57:54.161265 6352 tablet_replica.cc:331] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:54.164597 6352 raft_consensus.cc:2241] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250626 19:57:54.165980 4756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.131:40711, user_credentials={real_user=slave}} blocked reactor thread for 62499us
W20250626 19:57:54.166705 4756 leader_election.cc:343] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:54.167183 4756 leader_election.cc:304] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f885c6802341c889a40e9c20fbfff1; no voters: 4d3be519a0044a2f921a503412f72e86, 5f8fb6d535ea42359fe34478df9e00a6
I20250626 19:57:54.167992 6121 raft_consensus.cc:2747] T 11648a3a9a94450284b8bc14ffe95852 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:57:54.168516 5366 raft_consensus.cc:738] T f83ef2b2b82b4fa4996b12ce3d8b5401 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:54.174041 5366 consensus_queue.cc:260] T f83ef2b2b82b4fa4996b12ce3d8b5401 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:54.176102 5366 ts_tablet_manager.cc:1428] T f83ef2b2b82b4fa4996b12ce3d8b5401 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20250626 19:57:54.180241 6121 raft_consensus.cc:491] T f83ef2b2b82b4fa4996b12ce3d8b5401 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:54.180727 6121 raft_consensus.cc:513] T f83ef2b2b82b4fa4996b12ce3d8b5401 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:54.182866 6121 leader_election.cc:290] T f83ef2b2b82b4fa4996b12ce3d8b5401 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711), 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845)
I20250626 19:57:54.190616 4396 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
W20250626 19:57:54.170049 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 79533.7us
I20250626 19:57:54.194563 4396 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 8f1a296c114b4149bc694f6b08a1f2e9 (table default.Table_1f95cb776ca24388979785af3b4e84c6 [id=9e99b7ec61654e36830933bab686b355]). Expected version 1 got 0
I20250626 19:57:54.196223 4396 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (table default.Table_56852b6e38af4545879ff672cc528975 [id=e3400f897e9944549162854e3f92ac21]). Expected version 1 got 0
I20250626 19:57:54.172609 5353 tablet_bootstrap.cc:492] T 56f319c748374b7cbbf037ef55aecd1b P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:54.199064 5353 ts_tablet_manager.cc:1397] T 56f319c748374b7cbbf037ef55aecd1b P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.000s
I20250626 19:57:54.206074 5353 raft_consensus.cc:357] T 56f319c748374b7cbbf037ef55aecd1b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:54.206944 5353 raft_consensus.cc:383] T 56f319c748374b7cbbf037ef55aecd1b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:54.207252 5353 raft_consensus.cc:738] T 56f319c748374b7cbbf037ef55aecd1b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:54.207978 5353 consensus_queue.cc:260] T 56f319c748374b7cbbf037ef55aecd1b P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:54.191447 4972 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f83ef2b2b82b4fa4996b12ce3d8b5401" candidate_uuid: "93f885c6802341c889a40e9c20fbfff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8fb6d535ea42359fe34478df9e00a6" is_pre_election: true
I20250626 19:57:54.198074 4396 catalog_manager.cc:5582] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:54.218258 5353 ts_tablet_manager.cc:1428] T 56f319c748374b7cbbf037ef55aecd1b P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250626 19:57:54.226178 4396 catalog_manager.cc:5582] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:54.227752 4396 catalog_manager.cc:5582] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:54.229265 4396 catalog_manager.cc:5582] T 83f0431638144710b598b3f87d9a576e P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } }
I20250626 19:57:54.230686 4396 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 27acc0eafbb04375ba1c6a59e958e6f6 (table default.Table_2d1ec663bb8048e0ac15d7025383ef5d [id=30ad5642ec2d422aad5b97698297fbe9]). Expected version 1 got 0
I20250626 19:57:54.231774 4396 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac]). Expected version 1 got 0
I20250626 19:57:54.232518 4396 catalog_manager.cc:5582] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:54.234696 4396 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 1b794133c1a346318e31bffb4ba6effb (table default.Table_b8d28650a552496b9582e1623ea694ff [id=574f9c9a69d34e918b567ac099e081e5]). Expected version 1 got 0
I20250626 19:57:54.235494 5353 tablet_bootstrap.cc:492] T 8f54a2d0098446a5b60dc6d07a234bc4 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:54.235636 4396 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 34b2919e5f534f4295336a4384be1f64 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:54.236100 4396 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:54.239491 4396 catalog_manager.cc:5582] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:54.240633 6352 raft_consensus.cc:2270] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:54.243811 5366 tablet_bootstrap.cc:492] T 8c4549e621504a30ba5be5b0f139c905 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:54.245333 6353 tablet_replica.cc:331] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:54.246299 6353 raft_consensus.cc:2241] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:54.246901 6353 raft_consensus.cc:2270] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:54.248798 6353 ts_tablet_manager.cc:1905] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:54.250049 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 54e68eba747e4a2a947e84e53a444d08 (table default.Table_8ef072e9d240402397d79db39729e650 [id=211696a0b141485ab4e4e2b1d190baa4]) successfully deleted
I20250626 19:57:54.279601 6352 ts_tablet_manager.cc:1905] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:54.286216 5366 tablet_bootstrap.cc:654] T 8c4549e621504a30ba5be5b0f139c905 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:54.304247 5353 tablet_bootstrap.cc:654] T 8f54a2d0098446a5b60dc6d07a234bc4 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:54.305359 4756 leader_election.cc:343] T f83ef2b2b82b4fa4996b12ce3d8b5401 P 93f885c6802341c889a40e9c20fbfff1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:57:54.319039 5366 tablet_bootstrap.cc:492] T 8c4549e621504a30ba5be5b0f139c905 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:54.319487 5366 ts_tablet_manager.cc:1397] T 8c4549e621504a30ba5be5b0f139c905 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.000s
I20250626 19:57:54.321662 5366 raft_consensus.cc:357] T 8c4549e621504a30ba5be5b0f139c905 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:54.322278 5366 raft_consensus.cc:383] T 8c4549e621504a30ba5be5b0f139c905 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:54.322542 5366 raft_consensus.cc:738] T 8c4549e621504a30ba5be5b0f139c905 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:54.331732 5366 consensus_queue.cc:260] T 8c4549e621504a30ba5be5b0f139c905 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:54.335053 4393 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:54.339296 4393 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 8f1a296c114b4149bc694f6b08a1f2e9 (table default.Table_1f95cb776ca24388979785af3b4e84c6 [id=9e99b7ec61654e36830933bab686b355]). Expected version 1 got 0
I20250626 19:57:54.340696 5366 ts_tablet_manager.cc:1428] T 8c4549e621504a30ba5be5b0f139c905 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20250626 19:57:54.340636 4393 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (table default.Table_56852b6e38af4545879ff672cc528975 [id=e3400f897e9944549162854e3f92ac21]). Expected version 1 got 0
I20250626 19:57:54.341675 5366 tablet_bootstrap.cc:492] T 8c5c000402db45e182a551cd80626665 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:54.343981 4393 catalog_manager.cc:5582] T 27acc0eafbb04375ba1c6a59e958e6f6 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:54.345170 4393 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 27acc0eafbb04375ba1c6a59e958e6f6 (table default.Table_2d1ec663bb8048e0ac15d7025383ef5d [id=30ad5642ec2d422aad5b97698297fbe9]). Expected version 1 got 0
I20250626 19:57:54.346164 4393 catalog_manager.cc:5582] T b852dd204f8a42cca629901723fc9247 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:54.347589 4393 catalog_manager.cc:5582] T 4a6058961bc7450ab489b26158e7dd17 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:54.349177 4393 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac]). Expected version 1 got 0
I20250626 19:57:54.350421 4393 catalog_manager.cc:5582] T 075dbe46bab14d40a8008ce5c24bcf38 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3be519a0044a2f921a503412f72e86 (127.4.46.129). New cstate: current_term: 1 leader_uuid: "4d3be519a0044a2f921a503412f72e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:54.351940 4393 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 1b794133c1a346318e31bffb4ba6effb (table default.Table_b8d28650a552496b9582e1623ea694ff [id=574f9c9a69d34e918b567ac099e081e5]). Expected version 1 got 0
I20250626 19:57:54.352535 4393 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet 34b2919e5f534f4295336a4384be1f64 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:54.354508 4393 catalog_manager.cc:5582] T de0724ba71824a5daaf6f3708b4431b7 P 4d3be519a0044a2f921a503412f72e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 1 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } }
I20250626 19:57:54.355517 4393 catalog_manager.cc:5674] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:54.373281 5366 tablet_bootstrap.cc:654] T 8c5c000402db45e182a551cd80626665 P 93f885c6802341c889a40e9c20fbfff1: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:54.424120 6353 ts_tablet_manager.cc:1918] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250626 19:57:54.429554 4383 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 7851e464fcd44ecb82404d6a991db2a2 (table default.Table_35dcbe7e914846d8b6b52efa647c4d98 [id=91dc6e79ab2b4c4badbee78be21918fd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7851e464fcd44ecb82404d6a991db2a2 consensus state has no leader
I20250626 19:57:54.466696 5025 heartbeater.cc:499] Master 127.4.46.189:45039 was elected leader, sending a full tablet report...
I20250626 19:57:54.426609 6353 log.cc:1199] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/56f319c748374b7cbbf037ef55aecd1b
I20250626 19:57:54.481607 6353 ts_tablet_manager.cc:1939] T 56f319c748374b7cbbf037ef55aecd1b P 93f885c6802341c889a40e9c20fbfff1: Deleting consensus metadata
I20250626 19:57:54.484233 5353 tablet_bootstrap.cc:492] T 8f54a2d0098446a5b60dc6d07a234bc4 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:54.484896 5353 ts_tablet_manager.cc:1397] T 8f54a2d0098446a5b60dc6d07a234bc4 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.250s user 0.015s sys 0.007s
I20250626 19:57:54.502171 5366 tablet_bootstrap.cc:492] T 8c5c000402db45e182a551cd80626665 P 93f885c6802341c889a40e9c20fbfff1: No bootstrap required, opened a new log
I20250626 19:57:54.502784 5366 ts_tablet_manager.cc:1397] T 8c5c000402db45e182a551cd80626665 P 93f885c6802341c889a40e9c20fbfff1: Time spent bootstrapping tablet: real 0.161s user 0.023s sys 0.004s
I20250626 19:57:54.505694 5366 raft_consensus.cc:357] T 8c5c000402db45e182a551cd80626665 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:54.506286 5366 raft_consensus.cc:383] T 8c5c000402db45e182a551cd80626665 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:54.507081 5366 raft_consensus.cc:738] T 8c5c000402db45e182a551cd80626665 P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f885c6802341c889a40e9c20fbfff1, State: Initialized, Role: FOLLOWER
I20250626 19:57:54.507664 5366 consensus_queue.cc:260] T 8c5c000402db45e182a551cd80626665 P 93f885c6802341c889a40e9c20fbfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:54.507488 5353 raft_consensus.cc:357] T 8f54a2d0098446a5b60dc6d07a234bc4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:54.508437 5353 raft_consensus.cc:383] T 8f54a2d0098446a5b60dc6d07a234bc4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:54.508723 5353 raft_consensus.cc:738] T 8f54a2d0098446a5b60dc6d07a234bc4 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:54.509413 5353 consensus_queue.cc:260] T 8f54a2d0098446a5b60dc6d07a234bc4 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
I20250626 19:57:54.510668 4391 catalog_manager.cc:5582] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } }
I20250626 19:57:54.521035 4391 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (table default.Table_56852b6e38af4545879ff672cc528975 [id=e3400f897e9944549162854e3f92ac21]). Expected version 1 got 0
I20250626 19:57:54.522502 4391 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac]). Expected version 1 got 0
W20250626 19:57:54.522651 4911 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:39951, user_credentials={real_user=slave}} blocked reactor thread for 142368us
I20250626 19:57:54.523166 6427 raft_consensus.cc:491] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:54.523823 6427 raft_consensus.cc:513] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:54.524624 4391 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:54.525709 4391 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 27acc0eafbb04375ba1c6a59e958e6f6 (table default.Table_2d1ec663bb8048e0ac15d7025383ef5d [id=30ad5642ec2d422aad5b97698297fbe9]). Expected version 1 got 0
I20250626 19:57:54.526155 5353 ts_tablet_manager.cc:1428] T 8f54a2d0098446a5b60dc6d07a234bc4 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.041s user 0.008s sys 0.000s
I20250626 19:57:54.526233 4391 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 1b794133c1a346318e31bffb4ba6effb (table default.Table_b8d28650a552496b9582e1623ea694ff [id=574f9c9a69d34e918b567ac099e081e5]). Expected version 1 got 0
I20250626 19:57:54.527388 5353 tablet_bootstrap.cc:492] T 34b2919e5f534f4295336a4384be1f64 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:54.539733 6427 leader_election.cc:290] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
I20250626 19:57:54.541286 5353 tablet_bootstrap.cc:654] T 34b2919e5f534f4295336a4384be1f64 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
I20250626 19:57:54.546830 4383 catalog_manager.cc:4928] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): tablet 56f319c748374b7cbbf037ef55aecd1b (table default.Table_bf54a23958c14a1699f712665f29c0ad [id=a1cfdad9d9584a04be9e39bb35c60fdf]) successfully deleted
W20250626 19:57:54.548254 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet e7c352224d0a46c6a922bf0a0d765f17 because the tablet was not found. No further retry: Not found: Tablet not found: e7c352224d0a46c6a922bf0a0d765f17
W20250626 19:57:54.549968 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet f807cf54bbb146ff989762c18410e6bb because the tablet was not found. No further retry: Not found: Tablet not found: f807cf54bbb146ff989762c18410e6bb
W20250626 19:57:54.551626 4383 catalog_manager.cc:4895] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951): delete failed for tablet d8be634cb36d40dca3f9338dab93ad2c because the tablet was not found. No further retry: Not found: Tablet not found: d8be634cb36d40dca3f9338dab93ad2c
I20250626 19:57:54.553020 6353 tablet_replica.cc:331] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1: stopping tablet replica
I20250626 19:57:54.554479 6353 raft_consensus.cc:2241] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:57:54.556515 6353 raft_consensus.cc:2270] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:54.559406 5366 ts_tablet_manager.cc:1428] T 8c5c000402db45e182a551cd80626665 P 93f885c6802341c889a40e9c20fbfff1: Time spent starting tablet: real 0.056s user 0.005s sys 0.001s
I20250626 19:57:54.571722 5366 tablet_bootstrap.cc:492] T ddbc235ec29d41d28af358b96099f8d0 P 93f885c6802341c889a40e9c20fbfff1: Bootstrap starting.
I20250626 19:57:54.577019 4391 catalog_manager.cc:5582] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "5f8fb6d535ea42359fe34478df9e00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:54.584446 6352 ts_tablet_manager.cc:1918] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:54.584961 6352 log.cc:1199] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/f807cf54bbb146ff989762c18410e6bb
I20250626 19:57:54.585973 6352 ts_tablet_manager.cc:1939] T f807cf54bbb146ff989762c18410e6bb P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:54.593881 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet f807cf54bbb146ff989762c18410e6bb (table default.Table_f85ad493ebbf4e3b8c7eea1edfcf6a67 [id=fa2e7940491b4fbbac3ee7d93df1476d]) successfully deleted
W20250626 19:57:54.603101 4384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.46.129:39845 (error 108) [suppressed 4 similar messages]
W20250626 19:57:54.603250 4754 connection.cc:537] server connection from 127.4.46.129:42661 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 19:57:54.603790 4755 connection.cc:537] client connection to 127.4.46.129:39845 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 19:57:54.604275 4755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250626 19:57:54.604750 5353 tablet_bootstrap.cc:492] T 34b2919e5f534f4295336a4384be1f64 P 5f8fb6d535ea42359fe34478df9e00a6: No bootstrap required, opened a new log
I20250626 19:57:54.605136 5353 ts_tablet_manager.cc:1397] T 34b2919e5f534f4295336a4384be1f64 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.020s
I20250626 19:57:54.606738 6352 tablet_replica.cc:331] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
I20250626 19:57:54.608363 6352 raft_consensus.cc:2241] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:54.608430 5353 raft_consensus.cc:357] T 34b2919e5f534f4295336a4384be1f64 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:54.609091 5353 raft_consensus.cc:383] T 34b2919e5f534f4295336a4384be1f64 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:57:54.609053 6352 raft_consensus.cc:2270] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:54.609401 5353 raft_consensus.cc:738] T 34b2919e5f534f4295336a4384be1f64 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8fb6d535ea42359fe34478df9e00a6, State: Initialized, Role: FOLLOWER
I20250626 19:57:54.610859 4391 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet 34b2919e5f534f4295336a4384be1f64 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:54.612442 4391 catalog_manager.cc:5674] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711) does not have the latest schema for tablet c2ff9841bbbd429b83c123e1ebdabfea (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:54.613596 4815 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d0a30ee16d043f9837dce612dae670a" candidate_uuid: "5f8fb6d535ea42359fe34478df9e00a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f885c6802341c889a40e9c20fbfff1" is_pre_election: true
I20250626 19:57:54.614210 4815 raft_consensus.cc:2466] T 2d0a30ee16d043f9837dce612dae670a P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8fb6d535ea42359fe34478df9e00a6 in term 0.
I20250626 19:57:54.615401 6427 consensus_queue.cc:1035] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:57:54.615900 4912 connection.cc:700] client connection to 127.4.46.129:39845 send error: Network error: sendmsg error: Connection reset by peer (error 104)
W20250626 19:57:54.620692 4912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: sendmsg error: Connection reset by peer (error 104)
I20250626 19:57:54.577498 6353 ts_tablet_manager.cc:1905] T 419c4018256546ef9431ac2db619ee33 P 93f885c6802341c889a40e9c20fbfff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:54.624228 4282 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu with pid 4725
I20250626 19:57:54.624197 6121 raft_consensus.cc:491] T 9e4673214ad84f75a861c3af82a13d8f P 93f885c6802341c889a40e9c20fbfff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:54.623208 5353 consensus_queue.cc:260] T 34b2919e5f534f4295336a4384be1f64 P 5f8fb6d535ea42359fe34478df9e00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
I20250626 19:57:54.639286 5353 ts_tablet_manager.cc:1428] T 34b2919e5f534f4295336a4384be1f64 P 5f8fb6d535ea42359fe34478df9e00a6: Time spent starting tablet: real 0.034s user 0.004s sys 0.002s
I20250626 19:57:54.640338 5353 tablet_bootstrap.cc:492] T 7def0bb1dbdc494eaa71748ebe1f8847 P 5f8fb6d535ea42359fe34478df9e00a6: Bootstrap starting.
I20250626 19:57:54.640421 6352 ts_tablet_manager.cc:1905] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:57:54.628820 4911 leader_election.cc:304] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6, 93f885c6802341c889a40e9c20fbfff1; no voters:
I20250626 19:57:54.643252 6426 raft_consensus.cc:2802] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:57:54.643602 6426 raft_consensus.cc:491] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:57:54.644001 6426 raft_consensus.cc:3058] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:57:54.646865 4384 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.46.129:39845, user_credentials={real_user=slave}} blocked reactor thread for 43801.8us
I20250626 19:57:54.651468 6427 consensus_queue.cc:1035] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:57:54.664901 4912 connection.cc:537] server connection from 127.4.46.129:45277 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 19:57:54.668009 4912 consensus_peers.cc:487] T 6c86ced5a3dc48e380244a9548d2f4c8 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.668478 4912 consensus_peers.cc:487] T cf5c4cd6c2f64877a9e348dedb9820a3 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.668846 4912 consensus_peers.cc:487] T 8ea7179912284fef8d2165cdfaf4a21a P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.672001 4384 catalog_manager.cc:4671] TS 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e94a73b50b564a97bbee0f5015bb8805: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111)
W20250626 19:57:54.669181 4912 consensus_peers.cc:487] T e0bcbb044d1549c88e5b83eeb5c709af P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.672837 4912 consensus_peers.cc:487] T 6d43617ed2cb4a0085e59eefcf8b19f1 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.673224 4912 consensus_peers.cc:487] T a3b05549a1434a1fa3d337fdfb96a131 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.673590 4912 consensus_peers.cc:487] T 5bf4c4596b4441839c82b5fe5c5eea37 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.674042 4912 consensus_peers.cc:487] T d7c5aba053ca4535ad2376b2066cfe47 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.674387 4912 consensus_peers.cc:487] T 6740fcfd3d41487b8a4cb2a6af1e70d4 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.684801 4912 consensus_peers.cc:487] T fafbf98e7f4544b6ad96c1c5ab6c82bc P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.685222 4912 consensus_peers.cc:487] T 0f834922c9de46dca48c47748913b11a P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.685562 4912 consensus_peers.cc:487] T f70c620bf4474e59aa1f86761aca0daa P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.685894 4912 consensus_peers.cc:487] T d668a84475064a3eaa08b0c719471c26 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.686251 4912 leader_election.cc:336] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111)
W20250626 19:57:54.686915 4912 consensus_peers.cc:487] T 4fb3692f4ddb4ff89c1817da4a5cb927 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:54.690052 6427 raft_consensus.cc:491] T 11648a3a9a94450284b8bc14ffe95852 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:57:54.689976 6426 raft_consensus.cc:513] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } } peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } }
W20250626 19:57:54.690548 4912 consensus_peers.cc:487] T 6cc7a336385f499ca19566d5f2901e98 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:54.690526 6427 raft_consensus.cc:513] T 11648a3a9a94450284b8bc14ffe95852 P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } }
W20250626 19:57:54.691068 4912 consensus_peers.cc:487] T 6fdf32e1756447c9a9e59956bb18b772 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:54.691948 6426 leader_election.cc:290] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:54.691421 4912 consensus_peers.cc:487] T 8282e22bfb8b4d4994fa8b95ef154a68 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:54.692329 6427 leader_election.cc:290] T 11648a3a9a94450284b8bc14ffe95852 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845), 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951)
W20250626 19:57:54.692590 4912 leader_election.cc:336] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111)
I20250626 19:57:54.705147 4912 leader_election.cc:304] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8fb6d535ea42359fe34478df9e00a6; no voters: 4d3be519a0044a2f921a503412f72e86, 93f885c6802341c889a40e9c20fbfff1
I20250626 19:57:54.705968 6427 raft_consensus.cc:2747] T e953f25481a245dd8edd61cd2e6061d1 P 5f8fb6d535ea42359fe34478df9e00a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:57:54.706287 4912 consensus_peers.cc:487] T 159a6bf746e14479b3f3584640b63098 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.706691 4912 consensus_peers.cc:487] T e9f09f0381eb420ca57c18a5be6f2cc2 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.707098 4912 consensus_peers.cc:487] T 61fcd55f6d7b4a369383d4f73d5f483d P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.711925 4912 consensus_peers.cc:487] T d8399c219d0e4485bb7926eefd78f210 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.712330 4912 consensus_peers.cc:487] T 44960f8db08942879cde6f96e9c620da P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.714874 4912 leader_election.cc:336] T 2d0a30ee16d043f9837dce612dae670a P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111)
W20250626 19:57:54.715304 4912 leader_election.cc:336] T 11648a3a9a94450284b8bc14ffe95852 P 5f8fb6d535ea42359fe34478df9e00a6 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111)
W20250626 19:57:54.715703 4912 consensus_peers.cc:487] T 465ed0fe56fb4960a58e92b9b3d594e6 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.719141 4912 consensus_peers.cc:487] T 1474bfa74a6346bf9e3ff49019f6c9b3 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:54.750124 5353 tablet_bootstrap.cc:654] T 7def0bb1dbdc494eaa71748ebe1f8847 P 5f8fb6d535ea42359fe34478df9e00a6: Neither blocks nor log segments found. Creating new log.
W20250626 19:57:54.752610 4912 consensus_peers.cc:487] T 7aee4118595346ef990a8a52eb007072 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 19:57:54.755455 4912 consensus_peers.cc:487] T 0e6136b0d3104ae88c3f4ae88a41de84 P 5f8fb6d535ea42359fe34478df9e00a6 -> Peer 4d3be519a0044a2f921a503412f72e86 (127.4.46.129:39845): Couldn't send request to peer 4d3be519a0044a2f921a503412f72e86. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.129:39845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 19:57:54.765354 6352 ts_tablet_manager.cc:1918] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:57:54.771571 6352 log.cc:1199] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/1feb6785d0c7459fbc6898b0c9d3e38c
I20250626 19:57:54.772532 6352 ts_tablet_manager.cc:1939] T 1feb6785d0c7459fbc6898b0c9d3e38c P 5f8fb6d535ea42359fe34478df9e00a6: Deleting consensus metadata
I20250626 19:57:54.777281 6352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 284) took 1036 ms. Trace:
W20250626 19:57:54.780273 4912 connection.cc:537] server connection from 127.4.46.130:60517 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250626 19:57:54.782903 6352 rpcz_store.cc:276] 0626 19:57:53.741090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:53.741204 (+ 114us) service_pool.cc:225] Handling call
0626 19:57:54.601092 (+859888us) ts_tablet_manager.cc:1145] Deleting tablet 1feb6785d0c7459fbc6898b0c9d3e38c
0626 19:57:54.606556 (+ 5464us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:54.693119 (+ 86563us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:54.765099 (+ 71980us) tablet_metadata.cc:641] Metadata flushed
0626 19:57:54.777211 (+ 12112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":859366}
I20250626 19:57:54.784476 6352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41356 (request call id 285) took 1042 ms. Trace:
I20250626 19:57:54.785161 4385 catalog_manager.cc:4928] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): tablet 1feb6785d0c7459fbc6898b0c9d3e38c (table default.Table_987278d268e04477828ad87c6d9c8c01 [id=da221881c13940a4b0dfeb88aa81c932]) successfully deleted
I20250626 19:57:54.784839 6352 rpcz_store.cc:276] 0626 19:57:53.742224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:57:53.742340 (+ 116us) service_pool.cc:225] Handling call
0626 19:57:54.784142 (+1041802us) ts_tablet_manager.cc:1145] Deleting tablet d8be634cb36d40dca3f9338dab93ad2c
0626 19:57:54.784200 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:57:54.784422 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1041285}
I20250626 19:57:54.785871 6352 tablet_replica.cc:331] T 56f319c748374b7cbbf037ef55aecd1b P 5f8fb6d535ea42359fe34478df9e00a6: stopping tablet replica
W20250626 19:57:54.787269 4385 catalog_manager.cc:4895] TS 5f8fb6d535ea42359fe34478df9e00a6 (127.4.46.131:40711): delete failed for tablet d8be634cb36d40dca3f9338dab93ad2c because the tablet was not found. No further retry: Not found: Tablet not found: d8be634cb36d40dca3f9338dab93ad2c
I20250626 19:57:54.789601 6352 raft_consensus.cc:2241] T 56f319c748374b7cbbf037ef55aecd1b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:57:54.790091 6352 raft_consensus.cc:2270] T 56f319c748374b7cbbf037ef55aecd1b P 5f8fb6d535ea42359fe34478df9e00a6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:57:54.791095 4282 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu with pid 4875
I20250626 19:57:54.806414 4394 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 5c113e03594b428a979435618fe373c6 (table default.Table_c022e4f4da8b477a8606e9c34729c8eb [id=3c86bc87a4a64fd8af0af50c516a2242]). Expected version 1 got 0
I20250626 19:57:54.808504 4394 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 8f1a296c114b4149bc694f6b08a1f2e9 (table default.Table_1f95cb776ca24388979785af3b4e84c6 [id=9e99b7ec61654e36830933bab686b355]). Expected version 1 got 0
I20250626 19:57:54.809597 4394 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (table default.Table_56852b6e38af4545879ff672cc528975 [id=e3400f897e9944549162854e3f92ac21]). Expected version 1 got 0
I20250626 19:57:54.812340 4394 catalog_manager.cc:5582] T e4aee96ad3454d65b974c867605c4cdc P 93f885c6802341c889a40e9c20fbfff1 reported cstate change: term changed from 0 to 2, leader changed from <none> to 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130). New cstate: current_term: 2 leader_uuid: "93f885c6802341c889a40e9c20fbfff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8fb6d535ea42359fe34478df9e00a6" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3be519a0044a2f921a503412f72e86" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f885c6802341c889a40e9c20fbfff1" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 39951 } health_report { overall_health: HEALTHY } } }
I20250626 19:57:54.814399 4394 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 27acc0eafbb04375ba1c6a59e958e6f6 (table default.Table_2d1ec663bb8048e0ac15d7025383ef5d [id=30ad5642ec2d422aad5b97698297fbe9]). Expected version 1 got 0
I20250626 19:57:54.816329 4394 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac]). Expected version 1 got 0
I20250626 19:57:54.818228 4394 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 1b794133c1a346318e31bffb4ba6effb (table default.Table_b8d28650a552496b9582e1623ea694ff [id=574f9c9a69d34e918b567ac099e081e5]). Expected version 1 got 0
I20250626 19:57:54.819108 4394 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 34b2919e5f534f4295336a4384be1f64 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:54.819787 4394 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
W20250626 19:57:54.956197 5770 connection.cc:537] server connection from 127.4.46.131:44819 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 19:57:54.957417 4385 connection.cc:537] client connection to 127.4.46.131:40711 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250626 19:57:54.966820 4282 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu with pid 5266
I20250626 19:57:54.996197 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 7def0bb1dbdc494eaa71748ebe1f8847 (table default.Table_56852b6e38af4545879ff672cc528975 [id=e3400f897e9944549162854e3f92ac21]). Expected version 1 got 0
I20250626 19:57:54.997495 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet e9f09f0381eb420ca57c18a5be6f2cc2 (table default.Table_9b87a72676524807b2c26f78807b3d9d [id=f588a91106544d889257bebe79a0a7ac]). Expected version 1 got 0
I20250626 19:57:54.998844 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 27acc0eafbb04375ba1c6a59e958e6f6 (table default.Table_2d1ec663bb8048e0ac15d7025383ef5d [id=30ad5642ec2d422aad5b97698297fbe9]). Expected version 1 got 0
I20250626 19:57:55.001911 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 34b2919e5f534f4295336a4384be1f64 (table default.Table_94a9ea25bac540f59bd2e2b6beb127c1 [id=46d6d30275b2427a8591d9d3d6101194]). Expected version 2 got 0
I20250626 19:57:55.003088 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet c2ff9841bbbd429b83c123e1ebdabfea (table default.Table_ffa0f924726d484ab1a91bf90ed84f93 [id=3af53a80de594b7a99fa10334f77864d]). Expected version 1 got 0
I20250626 19:57:55.004379 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet 1b794133c1a346318e31bffb4ba6effb (table default.Table_b8d28650a552496b9582e1623ea694ff [id=574f9c9a69d34e918b567ac099e081e5]). Expected version 1 got 0
I20250626 19:57:55.005666 4390 catalog_manager.cc:5674] TS 93f885c6802341c889a40e9c20fbfff1 (127.4.46.130:39951) does not have the latest schema for tablet de0724ba71824a5daaf6f3708b4431b7 (table default.Table_c98cdc17c95f4f1295e667b2c8324085 [id=629ab240ba9a40e3a581b74ae5ed4f71]). Expected version 1 got 0
I20250626 19:57:55.034678 4282 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu with pid 4361
I20250626 19:57:55.083554 4282 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu with pid 4436
2025-06-26T19:57:55Z chronyd exiting
I20250626 19:57:55.131773 4282 master-stress-test.cc:585] Tables created: 130
I20250626 19:57:55.131973 4282 master-stress-test.cc:586] Tables altered: 37
I20250626 19:57:55.132095 4282 master-stress-test.cc:587] Tables deleted: 32
I20250626 19:57:55.132212 4282 master-stress-test.cc:588] Tablets replaced: 27
I20250626 19:57:55.132321 4282 master-stress-test.cc:589] Masters restarted: 1
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (44727 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
I20250626 19:57:55.221370 4282 test_util.cc:276] Using random seed: -97044637
2025-06-26T19:57:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T19:57:55Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskAJMAmr/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskAJMAmr/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/security.properties
2025-06-26 19:58:01: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskAJMAmr/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskAJMAmr/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250626 19:58:11.543635 4282 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
/tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.190:39311
--webserver_interface=127.4.46.190
--webserver_port=0
--builtin_ntp_servers=127.4.46.148:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919
--hive_metastore_uris=thrift://127.0.0.1:33633
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250626 19:58:11.835412 6685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 19:58:11.835961 6685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 19:58:11.836382 6685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 19:58:11.867578 6685 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250626 19:58:11.867832 6685 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250626 19:58:11.868033 6685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 19:58:11.868232 6685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 19:58:11.868434 6685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 19:58:11.868625 6685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 19:58:11.905229 6685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.46.148:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:33633
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.190:39311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.46.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6785
TSAN enabled
I20250626 19:58:11.906551 6685 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 19:58:11.908186 6685 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 19:58:11.923915 6692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:58:11.924125 6691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:58:11.923903 6694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:58:13.107439 6693 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 19:58:13.107486 6685 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 19:58:13.111696 6685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 19:58:13.114207 6685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 19:58:13.115562 6685 hybrid_clock.cc:648] HybridClock initialized: now 1750967893115550 us; error 49 us; skew 500 ppm
I20250626 19:58:13.116408 6685 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 19:58:13.123152 6685 webserver.cc:469] Webserver started at http://127.4.46.190:33659/ using document root <none> and password file <none>
I20250626 19:58:13.124059 6685 fs_manager.cc:362] Metadata directory not provided
I20250626 19:58:13.124284 6685 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 19:58:13.124727 6685 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 19:58:13.129153 6685 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/data/instance:
uuid: "188d02ec81e645229acdc5eed98c9f9f"
format_stamp: "Formatted at 2025-06-26 19:58:13 on dist-test-slave-48dm"
I20250626 19:58:13.130240 6685 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/wal/instance:
uuid: "188d02ec81e645229acdc5eed98c9f9f"
format_stamp: "Formatted at 2025-06-26 19:58:13 on dist-test-slave-48dm"
I20250626 19:58:13.137452 6685 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250626 19:58:13.142935 6701 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 19:58:13.143924 6685 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250626 19:58:13.144251 6685 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/data,/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/wal
uuid: "188d02ec81e645229acdc5eed98c9f9f"
format_stamp: "Formatted at 2025-06-26 19:58:13 on dist-test-slave-48dm"
I20250626 19:58:13.144593 6685 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 19:58:13.199362 6685 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 19:58:13.200960 6685 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 19:58:13.201396 6685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 19:58:13.270803 6685 rpc_server.cc:307] RPC server started. Bound to: 127.4.46.190:39311
I20250626 19:58:13.270941 6752 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.46.190:39311 every 8 connection(s)
I20250626 19:58:13.273432 6685 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/data/info.pb
I20250626 19:58:13.278133 6753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:13.281635 4282 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu as pid 6685
I20250626 19:58:13.282042 4282 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-0/wal/instance
I20250626 19:58:13.284451 4282 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
/tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.189:36417
--webserver_interface=127.4.46.189
--webserver_port=0
--builtin_ntp_servers=127.4.46.148:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919
--hive_metastore_uris=thrift://127.0.0.1:33633
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250626 19:58:13.288779 6753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } has no permanent_uuid. Determining permanent_uuid...
I20250626 19:58:13.313686 6753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } has no permanent_uuid. Determining permanent_uuid...
W20250626 19:58:13.316087 6704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 19:58:13.320343 6753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.189:36417: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
I20250626 19:58:13.369822 6753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } attempt: 1
W20250626 19:58:13.374405 6753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.189:36417: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
I20250626 19:58:13.442845 6753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } attempt: 2
W20250626 19:58:13.446846 6753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.189:36417: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
I20250626 19:58:13.538232 6753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } attempt: 3
W20250626 19:58:13.541905 6753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.189:36417: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 19:58:13.585772 6754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 19:58:13.586377 6754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 19:58:13.586908 6754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 19:58:13.617678 6754 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250626 19:58:13.617985 6754 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250626 19:58:13.618252 6754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 19:58:13.618498 6754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 19:58:13.618736 6754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 19:58:13.619007 6754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 19:58:13.653972 6754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.46.148:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:33633
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.189:36417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.46.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6785
TSAN enabled
I20250626 19:58:13.655440 6754 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 19:58:13.657070 6754 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 19:58:13.672955 6764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:58:13.685292 6753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } attempt: 4
W20250626 19:58:13.689823 6753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.189:36417: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
I20250626 19:58:13.989243 6753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } attempt: 5
W20250626 19:58:13.995808 6753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.189:36417: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
I20250626 19:58:14.543408 6753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } attempt: 6
W20250626 19:58:14.554340 6753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.189:36417: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 19:58:13.672973 6763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:58:13.673159 6766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:58:14.871592 6765 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1196 milliseconds
I20250626 19:58:14.871708 6754 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 19:58:14.872934 6754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 19:58:14.876211 6754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 19:58:14.877614 6754 hybrid_clock.cc:648] HybridClock initialized: now 1750967894877584 us; error 46 us; skew 500 ppm
I20250626 19:58:14.878460 6754 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 19:58:14.885123 6754 webserver.cc:469] Webserver started at http://127.4.46.189:37313/ using document root <none> and password file <none>
I20250626 19:58:14.886003 6754 fs_manager.cc:362] Metadata directory not provided
I20250626 19:58:14.886188 6754 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 19:58:14.886736 6754 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 19:58:14.891220 6754 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data/instance:
uuid: "8ae38ec946e14198ae4adc8dde9d66e8"
format_stamp: "Formatted at 2025-06-26 19:58:14 on dist-test-slave-48dm"
I20250626 19:58:14.892326 6754 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/wal/instance:
uuid: "8ae38ec946e14198ae4adc8dde9d66e8"
format_stamp: "Formatted at 2025-06-26 19:58:14 on dist-test-slave-48dm"
I20250626 19:58:14.899448 6754 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250626 19:58:14.905009 6775 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 19:58:14.905999 6754 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250626 19:58:14.906307 6754 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data,/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/wal
uuid: "8ae38ec946e14198ae4adc8dde9d66e8"
format_stamp: "Formatted at 2025-06-26 19:58:14 on dist-test-slave-48dm"
I20250626 19:58:14.906651 6754 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 19:58:14.951875 6754 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 19:58:14.953334 6754 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 19:58:14.953759 6754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 19:58:15.023401 6754 rpc_server.cc:307] RPC server started. Bound to: 127.4.46.189:36417
I20250626 19:58:15.023435 6826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.46.189:36417 every 8 connection(s)
I20250626 19:58:15.025918 6754 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data/info.pb
I20250626 19:58:15.030694 6827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:15.033913 4282 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu as pid 6754
I20250626 19:58:15.034284 4282 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/wal/instance
I20250626 19:58:15.036522 4282 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
/tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.188:38919
--webserver_interface=127.4.46.188
--webserver_port=0
--builtin_ntp_servers=127.4.46.148:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919
--hive_metastore_uris=thrift://127.0.0.1:33633
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250626 19:58:15.041103 6827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } has no permanent_uuid. Determining permanent_uuid...
I20250626 19:58:15.061867 6827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } has no permanent_uuid. Determining permanent_uuid...
I20250626 19:58:15.076115 6827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } has no permanent_uuid. Determining permanent_uuid...
W20250626 19:58:15.078563 6779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
W20250626 19:58:15.082589 6827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:38919: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
I20250626 19:58:15.132130 6827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } attempt: 1
W20250626 19:58:15.136722 6827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:38919: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
I20250626 19:58:15.205080 6827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } attempt: 2
W20250626 19:58:15.209012 6827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:38919: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
I20250626 19:58:15.300488 6827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } attempt: 3
W20250626 19:58:15.304083 6827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:38919: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
W20250626 19:58:15.338300 6828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 19:58:15.338930 6828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 19:58:15.339406 6828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 19:58:15.370857 6828 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250626 19:58:15.371116 6828 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250626 19:58:15.371384 6828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 19:58:15.371646 6828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 19:58:15.371899 6828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 19:58:15.372136 6828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 19:58:15.407804 6828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.46.148:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:33633
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.188:38919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.46.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6785
TSAN enabled
I20250626 19:58:15.409162 6828 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 19:58:15.410773 6828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 19:58:15.424037 6838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:58:15.447619 6827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } attempt: 4
W20250626 19:58:15.451254 6827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:38919: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
I20250626 19:58:15.615012 6753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } attempt: 7
I20250626 19:58:15.627533 6753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } has no permanent_uuid. Determining permanent_uuid...
W20250626 19:58:15.632836 6753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:38919: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
I20250626 19:58:15.691468 6753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } attempt: 1
W20250626 19:58:15.695652 6753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:38919: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
I20250626 19:58:15.750953 6827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } attempt: 5
W20250626 19:58:15.755131 6827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:38919: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
I20250626 19:58:15.777299 6753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } attempt: 2
W20250626 19:58:15.781483 6753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:38919: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
I20250626 19:58:15.867089 6753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } attempt: 3
W20250626 19:58:15.871358 6753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:38919: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
I20250626 19:58:16.011986 6753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } attempt: 4
W20250626 19:58:16.016075 6753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:38919: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
I20250626 19:58:16.299687 6753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } attempt: 5
I20250626 19:58:16.302492 6827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } attempt: 6
W20250626 19:58:16.305620 6753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:38919: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
W20250626 19:58:16.311189 6827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.46.188:38919: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
W20250626 19:58:15.424871 6839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:58:15.426427 6841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:58:15.426964 6828 server_base.cc:1048] running on GCE node
I20250626 19:58:16.597606 6828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 19:58:16.600153 6828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 19:58:16.601505 6828 hybrid_clock.cc:648] HybridClock initialized: now 1750967896601479 us; error 63 us; skew 500 ppm
I20250626 19:58:16.602308 6828 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 19:58:16.608892 6828 webserver.cc:469] Webserver started at http://127.4.46.188:34669/ using document root <none> and password file <none>
I20250626 19:58:16.609817 6828 fs_manager.cc:362] Metadata directory not provided
I20250626 19:58:16.610024 6828 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 19:58:16.610515 6828 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 19:58:16.616017 6828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/data/instance:
uuid: "6610553ba0df459085f7373935774377"
format_stamp: "Formatted at 2025-06-26 19:58:16 on dist-test-slave-48dm"
I20250626 19:58:16.617107 6828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/wal/instance:
uuid: "6610553ba0df459085f7373935774377"
format_stamp: "Formatted at 2025-06-26 19:58:16 on dist-test-slave-48dm"
I20250626 19:58:16.624652 6828 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250626 19:58:16.629971 6853 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 19:58:16.631078 6828 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250626 19:58:16.631386 6828 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/data,/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/wal
uuid: "6610553ba0df459085f7373935774377"
format_stamp: "Formatted at 2025-06-26 19:58:16 on dist-test-slave-48dm"
I20250626 19:58:16.631703 6828 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 19:58:16.677484 6828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 19:58:16.678984 6828 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 19:58:16.679412 6828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 19:58:16.750166 6828 rpc_server.cc:307] RPC server started. Bound to: 127.4.46.188:38919
I20250626 19:58:16.750248 6904 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.46.188:38919 every 8 connection(s)
I20250626 19:58:16.752888 6828 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/data/info.pb
I20250626 19:58:16.755360 4282 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu as pid 6828
I20250626 19:58:16.755748 4282 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-2/wal/instance
I20250626 19:58:16.757927 6905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:16.773885 6905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } has no permanent_uuid. Determining permanent_uuid...
I20250626 19:58:16.793227 6905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } has no permanent_uuid. Determining permanent_uuid...
I20250626 19:58:16.803701 6905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } has no permanent_uuid. Determining permanent_uuid...
I20250626 19:58:16.819389 6905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: Bootstrap starting.
I20250626 19:58:16.826390 6905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:16.828179 6905 log.cc:826] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: Log is configured to *not* fsync() on all Append() calls
I20250626 19:58:16.832295 6905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: No bootstrap required, opened a new log
I20250626 19:58:16.850452 6905 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:16.851187 6905 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:16.851424 6905 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6610553ba0df459085f7373935774377, State: Initialized, Role: FOLLOWER
I20250626 19:58:16.852079 6905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:16.854413 6912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:16.855057 6912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:58:16.855643 6905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 19:58:16.858058 6753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } attempt: 6
I20250626 19:58:16.878610 6753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Bootstrap starting.
I20250626 19:58:16.885867 6753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:16.888743 6753 log.cc:826] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Log is configured to *not* fsync() on all Append() calls
I20250626 19:58:16.899820 6753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: No bootstrap required, opened a new log
I20250626 19:58:16.886250 6828 master_runner.cc:186] Error getting master registration for 127.4.46.190:39311: OK, instance_id { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" instance_seqno: 1750967893251898 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250626 19:58:16.914011 6828 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 19:58:16.913719 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:16.916563 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:16.926437 6753 raft_consensus.cc:357] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:16.927469 6753 raft_consensus.cc:383] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:16.927809 6753 raft_consensus.cc:738] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 188d02ec81e645229acdc5eed98c9f9f, State: Initialized, Role: FOLLOWER
I20250626 19:58:16.928887 6753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
W20250626 19:58:16.930703 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:16.931026 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:16.934191 6859 tablet.cc:2378] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 19:58:16.937106 6931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:16.940886 6931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:58:16.940570 6753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: configured and running, proceeding with master startup.
W20250626 19:58:16.944941 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:16.945329 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:16.959738 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:16.960193 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:16.960096 6707 tablet.cc:2378] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 19:58:16.974207 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:16.974619 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:16.988736 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:16.989161 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.003509 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.003906 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.018069 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.018503 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.034564 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.034996 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.049141 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.049561 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.063598 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.063977 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.078013 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.078469 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.092648 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.093158 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.094981 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.095417 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.056880 6685 master_runner.cc:186] Error getting master registration for 127.4.46.189:36417: OK, instance_id { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" instance_seqno: 1750967895004480 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250626 19:58:17.096179 6685 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 19:58:17.107575 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.107913 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.112039 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.112371 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.121996 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.122288 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.126199 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.126484 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.136350 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.136693 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.140513 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.140815 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.150570 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.150869 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.154560 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.154883 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.164870 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.165220 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.179793 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.180151 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.181051 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.181403 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.194124 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.194475 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.195189 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.195509 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.208853 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.209244 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.209580 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.209856 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.223618 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.223971 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.227293 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.227634 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.241902 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.241945 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.242226 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.242274 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.259022 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.259397 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.261310 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.261596 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.273373 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.273674 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.277558 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.278002 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.289346 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.293195 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.293604 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.293900 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.313419 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.314952 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.315304 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.315567 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.329525 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.329576 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.329890 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.329957 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.330567 6931 raft_consensus.cc:491] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:17.331063 6931 raft_consensus.cc:513] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:17.333427 6931 leader_election.cc:290] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
I20250626 19:58:17.335063 6880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "188d02ec81e645229acdc5eed98c9f9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6610553ba0df459085f7373935774377" is_pre_election: true
I20250626 19:58:17.335837 6880 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 188d02ec81e645229acdc5eed98c9f9f in term 0.
I20250626 19:58:17.337160 6705 leader_election.cc:304] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 188d02ec81e645229acdc5eed98c9f9f, 6610553ba0df459085f7373935774377; no voters:
I20250626 19:58:17.338076 6931 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:17.338435 6931 raft_consensus.cc:491] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:17.338800 6931 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:17.346294 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.346840 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.347371 6931 raft_consensus.cc:513] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:17.351558 6931 leader_election.cc:290] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 1 election: Requested vote from peers 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
I20250626 19:58:17.352540 6880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "188d02ec81e645229acdc5eed98c9f9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6610553ba0df459085f7373935774377"
I20250626 19:58:17.353078 6880 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:17.365760 6880 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 188d02ec81e645229acdc5eed98c9f9f in term 1.
I20250626 19:58:17.371426 6705 leader_election.cc:304] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 188d02ec81e645229acdc5eed98c9f9f, 6610553ba0df459085f7373935774377; no voters:
I20250626 19:58:17.371555 6827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } attempt: 7
I20250626 19:58:17.384915 6931 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 1 FOLLOWER]: Leader election won for term 1
W20250626 19:58:17.391870 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.392149 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.392026 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.396478 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.398703 6931 raft_consensus.cc:695] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 1 LEADER]: Becoming Leader. State: Replica: 188d02ec81e645229acdc5eed98c9f9f, State: Running, Role: LEADER
I20250626 19:58:17.399677 6931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
W20250626 19:58:17.408527 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.408797 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.417599 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.418035 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.424676 6950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 188d02ec81e645229acdc5eed98c9f9f. Latest consensus state: current_term: 1 leader_uuid: "188d02ec81e645229acdc5eed98c9f9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:17.425336 6950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: This master's current role is: LEADER
W20250626 19:58:17.425913 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250626 19:58:17.426609 6827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: Bootstrap starting.
W20250626 19:58:17.426496 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.434324 6945 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.434628 6945 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.441463 6952 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 19:58:17.443336 6827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:17.443703 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.444260 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.451122 6952 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 19:58:17.457351 6827 log.cc:826] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: Log is configured to *not* fsync() on all Append() calls
W20250626 19:58:17.462546 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.463346 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.464956 6827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: No bootstrap required, opened a new log
W20250626 19:58:17.483338 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.484007 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.493173 6880 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Refusing update from remote peer 188d02ec81e645229acdc5eed98c9f9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:58:17.494669 6950 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:17.504222 6827 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:17.505141 6827 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:17.505537 6827 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae38ec946e14198ae4adc8dde9d66e8, State: Initialized, Role: FOLLOWER
I20250626 19:58:17.506532 6827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:17.514568 6956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:17.518141 6956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:58:17.517979 6801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "188d02ec81e645229acdc5eed98c9f9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae38ec946e14198ae4adc8dde9d66e8"
I20250626 19:58:17.519039 6802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "188d02ec81e645229acdc5eed98c9f9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" is_pre_election: true
I20250626 19:58:17.517805 6827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 19:58:17.519096 6800 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:17.530709 6800 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Refusing update from remote peer 188d02ec81e645229acdc5eed98c9f9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 19:58:17.530966 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.531255 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.532425 6950 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:58:17.562968 6927 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 19:58:17.567497 6927 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.565004 6955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 188d02ec81e645229acdc5eed98c9f9f. Latest consensus state: current_term: 1 leader_uuid: "188d02ec81e645229acdc5eed98c9f9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:17.567888 6955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:58:17.572610 6954 mvcc.cc:204] Tried to move back new op lower bound from 7171964508104572928 to 7171964507758481408. Current Snapshot: MvccSnapshot[applied={T|T < 7171964508104572928}]
I20250626 19:58:17.573554 6956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 188d02ec81e645229acdc5eed98c9f9f. Latest consensus state: current_term: 1 leader_uuid: "188d02ec81e645229acdc5eed98c9f9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:17.574273 6956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:58:17.587124 6955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "188d02ec81e645229acdc5eed98c9f9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:17.587760 6955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:58:17.588405 6956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "188d02ec81e645229acdc5eed98c9f9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:17.589047 6956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:58:17.593513 6927 catalog_manager.cc:1261] Loaded cluster ID: 5cb1ee99f0744bcfa6ec1bff323536be
I20250626 19:58:17.593781 6927 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: loading cluster ID for follower catalog manager: success
I20250626 19:58:17.595213 6931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "188d02ec81e645229acdc5eed98c9f9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:17.595860 6931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: This master's current role is: LEADER
I20250626 19:58:17.596472 6931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "188d02ec81e645229acdc5eed98c9f9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:17.597056 6931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: This master's current role is: LEADER
W20250626 19:58:17.597342 6927 catalog_manager.cc:1578] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250626 19:58:17.597628 6927 catalog_manager.cc:875] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.599830 6952 catalog_manager.cc:1349] Generated new cluster ID: 5cb1ee99f0744bcfa6ec1bff323536be
I20250626 19:58:17.600103 6952 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
W20250626 19:58:17.615173 6927 catalog_manager.cc:1578] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250626 19:58:17.624891 6927 catalog_manager.cc:875] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.635185 6952 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 19:58:17.637079 6952 catalog_manager.cc:1506] Loading token signing keys...
W20250626 19:58:17.643962 6927 catalog_manager.cc:1578] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250626 19:58:17.644246 6927 catalog_manager.cc:875] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.661430 6927 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: acquiring CA information for follower catalog manager: success
W20250626 19:58:17.664718 6927 catalog_manager.cc:875] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20250626 19:58:17.689493 6927 catalog_manager.cc:875] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20250626 19:58:17.695186 6952 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Generated new TSK 0
I20250626 19:58:17.696242 6952 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 19:58:17.704695 6927 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 19:58:17.705576 6952 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250626 19:58:17.709592 6952 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250626 19:58:17.734308 4282 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
/tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.46.129:0
--local_ip_for_outbound_sockets=127.4.46.129
--webserver_interface=127.4.46.129
--webserver_port=0
--tserver_master_addrs=127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919
--builtin_ntp_servers=127.4.46.148:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250626 19:58:17.748973 6979 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250626 19:58:17.786564 6979 catalog_manager.cc:1261] Loaded cluster ID: 5cb1ee99f0744bcfa6ec1bff323536be
I20250626 19:58:17.786939 6979 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: loading cluster ID for follower catalog manager: success
I20250626 19:58:17.803839 6979 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: acquiring CA information for follower catalog manager: success
I20250626 19:58:17.832248 6979 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250626 19:58:18.467082 6980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 19:58:18.467799 6980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 19:58:18.468744 6980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 19:58:18.535391 6980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 19:58:18.537083 6980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.46.129
I20250626 19:58:18.612999 6980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.46.148:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.46.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.46.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.46.129
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6785
TSAN enabled
I20250626 19:58:18.615540 6980 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 19:58:18.617815 6980 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 19:58:18.666803 6987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:58:19.482358 6991 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:19.492480 6991 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:19.496040 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:19.499089 6880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6610553ba0df459085f7373935774377" is_pre_election: true
I20250626 19:58:19.500620 6779 leader_election.cc:304] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ae38ec946e14198ae4adc8dde9d66e8; no voters: 188d02ec81e645229acdc5eed98c9f9f, 6610553ba0df459085f7373935774377
I20250626 19:58:19.506821 6991 leader_election.cc:290] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
I20250626 19:58:19.511238 6991 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:58:20.030285 6993 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:20.030997 6993 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:20.034183 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:20.037321 6993 leader_election.cc:290] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417)
I20250626 19:58:20.037516 6801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" is_pre_election: true
I20250626 19:58:20.039461 6856 leader_election.cc:304] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6610553ba0df459085f7373935774377; no voters: 188d02ec81e645229acdc5eed98c9f9f, 8ae38ec946e14198ae4adc8dde9d66e8
I20250626 19:58:20.041510 6993 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:58:20.467069 6994 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:20.470984 6994 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:20.473737 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:20.479601 6880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6610553ba0df459085f7373935774377" is_pre_election: true
I20250626 19:58:20.481083 6779 leader_election.cc:304] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ae38ec946e14198ae4adc8dde9d66e8; no voters: 188d02ec81e645229acdc5eed98c9f9f, 6610553ba0df459085f7373935774377
I20250626 19:58:20.487255 6994 leader_election.cc:290] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
I20250626 19:58:20.488157 6994 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:58:20.645248 6996 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:20.645741 6996 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:20.649515 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:20.652889 6801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" is_pre_election: true
I20250626 19:58:20.654392 6856 leader_election.cc:304] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6610553ba0df459085f7373935774377; no voters: 188d02ec81e645229acdc5eed98c9f9f, 8ae38ec946e14198ae4adc8dde9d66e8
I20250626 19:58:20.662030 6996 leader_election.cc:290] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417)
I20250626 19:58:20.663707 6996 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 19:58:20.044559 6986 debug-util.cc:398] Leaking SignalData structure 0x7b08000186a0 after lost signal to thread 6980
W20250626 19:58:21.182160 6980 thread.cc:640] GCE (cloud detector) Time spent creating pthread: real 2.518s user 0.712s sys 1.331s
W20250626 19:58:18.667871 6988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:58:21.182534 6980 thread.cc:606] GCE (cloud detector) Time spent starting thread: real 2.518s user 0.712s sys 1.331s
W20250626 19:58:21.191883 6999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:58:21.191798 6980 server_base.cc:1048] running on GCE node
I20250626 19:58:21.193569 6980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 19:58:21.196480 6980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 19:58:21.198004 6980 hybrid_clock.cc:648] HybridClock initialized: now 1750967901197956 us; error 26 us; skew 500 ppm
I20250626 19:58:21.199142 6980 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 19:58:21.212491 6980 webserver.cc:469] Webserver started at http://127.4.46.129:33739/ using document root <none> and password file <none>
I20250626 19:58:21.213464 6980 fs_manager.cc:362] Metadata directory not provided
I20250626 19:58:21.213680 6980 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 19:58:21.214126 6980 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 19:58:21.218843 6980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/data/instance:
uuid: "bbe151be01e24c82854d9c8c79c797ca"
format_stamp: "Formatted at 2025-06-26 19:58:21 on dist-test-slave-48dm"
I20250626 19:58:21.219903 6980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/instance:
uuid: "bbe151be01e24c82854d9c8c79c797ca"
format_stamp: "Formatted at 2025-06-26 19:58:21 on dist-test-slave-48dm"
I20250626 19:58:21.227526 6980 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250626 19:58:21.233224 7005 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 19:58:21.234342 6980 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250626 19:58:21.234675 6980 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal
uuid: "bbe151be01e24c82854d9c8c79c797ca"
format_stamp: "Formatted at 2025-06-26 19:58:21 on dist-test-slave-48dm"
I20250626 19:58:21.235039 6980 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 19:58:21.291139 6980 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 19:58:21.293030 6980 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 19:58:21.293577 6980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 19:58:21.296849 6980 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 19:58:21.302795 6980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 19:58:21.303136 6980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250626 19:58:21.303439 6980 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 19:58:21.303644 6980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 19:58:21.443234 6980 rpc_server.cc:307] RPC server started. Bound to: 127.4.46.129:33129
I20250626 19:58:21.443346 7117 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.46.129:33129 every 8 connection(s)
I20250626 19:58:21.446250 6980 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/data/info.pb
I20250626 19:58:21.446853 4282 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu as pid 6980
I20250626 19:58:21.447423 4282 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/instance
I20250626 19:58:21.477749 4282 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
/tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.46.130:0
--local_ip_for_outbound_sockets=127.4.46.130
--webserver_interface=127.4.46.130
--webserver_port=0
--tserver_master_addrs=127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919
--builtin_ntp_servers=127.4.46.148:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250626 19:58:21.489234 7118 heartbeater.cc:344] Connected to a master server at 127.4.46.188:38919
I20250626 19:58:21.489774 7118 heartbeater.cc:461] Registering TS with master...
I20250626 19:58:21.491026 7120 heartbeater.cc:344] Connected to a master server at 127.4.46.190:39311
I20250626 19:58:21.491180 7118 heartbeater.cc:507] Master 127.4.46.188:38919 requested a full tablet report, sending...
I20250626 19:58:21.491394 7120 heartbeater.cc:461] Registering TS with master...
I20250626 19:58:21.492228 7120 heartbeater.cc:507] Master 127.4.46.190:39311 requested a full tablet report, sending...
I20250626 19:58:21.492988 7119 heartbeater.cc:344] Connected to a master server at 127.4.46.189:36417
I20250626 19:58:21.493356 7119 heartbeater.cc:461] Registering TS with master...
I20250626 19:58:21.494230 7119 heartbeater.cc:507] Master 127.4.46.189:36417 requested a full tablet report, sending...
I20250626 19:58:21.494719 6717 ts_manager.cc:194] Registered new tserver with Master: bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:21.494980 6870 ts_manager.cc:194] Registered new tserver with Master: bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:21.497195 6717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.46.129:56711
I20250626 19:58:21.497592 6792 ts_manager.cc:194] Registered new tserver with Master: bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
W20250626 19:58:21.808315 7128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 19:58:21.809049 7128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 19:58:21.810230 7128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 19:58:21.870005 7128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 19:58:21.870929 7128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.46.130
I20250626 19:58:21.910553 7128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.46.148:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.46.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.46.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.46.130
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6785
TSAN enabled
I20250626 19:58:21.911870 7128 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 19:58:21.913424 7128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 19:58:21.932672 7135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:58:21.945736 7140 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:21.946106 7140 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:21.948014 7140 leader_election.cc:290] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
I20250626 19:58:21.948566 6880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6610553ba0df459085f7373935774377" is_pre_election: true
I20250626 19:58:21.948459 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:21.950145 6777 leader_election.cc:304] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ae38ec946e14198ae4adc8dde9d66e8; no voters: 188d02ec81e645229acdc5eed98c9f9f, 6610553ba0df459085f7373935774377
I20250626 19:58:21.950834 7140 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:58:22.000396 7120 heartbeater.cc:499] Master 127.4.46.190:39311 was elected leader, sending a full tablet report...
I20250626 19:58:22.060220 7142 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:22.060600 7142 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:22.063926 7142 leader_election.cc:290] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417)
I20250626 19:58:22.064604 6801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" is_pre_election: true
I20250626 19:58:22.066262 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:22.067288 6855 leader_election.cc:304] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6610553ba0df459085f7373935774377; no voters: 188d02ec81e645229acdc5eed98c9f9f, 8ae38ec946e14198ae4adc8dde9d66e8
I20250626 19:58:22.067878 7142 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:58:22.653568 7143 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:22.653985 7143 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:22.655411 7143 leader_election.cc:290] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417)
I20250626 19:58:22.656389 6801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" is_pre_election: true
I20250626 19:58:22.656711 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:22.658418 6855 leader_election.cc:304] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6610553ba0df459085f7373935774377; no voters: 188d02ec81e645229acdc5eed98c9f9f, 8ae38ec946e14198ae4adc8dde9d66e8
I20250626 19:58:22.659104 7143 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 19:58:21.932664 7136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:58:21.934111 7138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 19:58:23.127596 7137 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 19:58:23.127702 7128 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 19:58:23.132416 7128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 19:58:23.135113 7128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 19:58:23.136511 7128 hybrid_clock.cc:648] HybridClock initialized: now 1750967903136473 us; error 56 us; skew 500 ppm
I20250626 19:58:23.137259 7128 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 19:58:23.143978 7128 webserver.cc:469] Webserver started at http://127.4.46.130:36327/ using document root <none> and password file <none>
I20250626 19:58:23.144846 7128 fs_manager.cc:362] Metadata directory not provided
I20250626 19:58:23.145051 7128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 19:58:23.145476 7128 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 19:58:23.149760 7128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/data/instance:
uuid: "315d304854c541bf8b5e6b1eea712d4a"
format_stamp: "Formatted at 2025-06-26 19:58:23 on dist-test-slave-48dm"
I20250626 19:58:23.150808 7128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/instance:
uuid: "315d304854c541bf8b5e6b1eea712d4a"
format_stamp: "Formatted at 2025-06-26 19:58:23 on dist-test-slave-48dm"
I20250626 19:58:23.157675 7128 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250626 19:58:23.163483 7148 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 19:58:23.164561 7128 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250626 19:58:23.164841 7128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal
uuid: "315d304854c541bf8b5e6b1eea712d4a"
format_stamp: "Formatted at 2025-06-26 19:58:23 on dist-test-slave-48dm"
I20250626 19:58:23.165127 7128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 19:58:23.216388 7128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 19:58:23.217759 7128 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 19:58:23.218186 7128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 19:58:23.220808 7128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 19:58:23.225198 7128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 19:58:23.225435 7128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 19:58:23.225669 7128 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 19:58:23.225824 7128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 19:58:23.368983 7128 rpc_server.cc:307] RPC server started. Bound to: 127.4.46.130:40373
I20250626 19:58:23.369118 7260 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.46.130:40373 every 8 connection(s)
I20250626 19:58:23.371490 7128 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/data/info.pb
I20250626 19:58:23.378657 4282 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu as pid 7128
I20250626 19:58:23.379354 4282 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/instance
I20250626 19:58:23.394357 4282 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
/tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.46.131:0
--local_ip_for_outbound_sockets=127.4.46.131
--webserver_interface=127.4.46.131
--webserver_port=0
--tserver_master_addrs=127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919
--builtin_ntp_servers=127.4.46.148:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250626 19:58:23.436139 7261 heartbeater.cc:344] Connected to a master server at 127.4.46.188:38919
I20250626 19:58:23.436626 7261 heartbeater.cc:461] Registering TS with master...
I20250626 19:58:23.437252 7262 heartbeater.cc:344] Connected to a master server at 127.4.46.189:36417
I20250626 19:58:23.437595 7262 heartbeater.cc:461] Registering TS with master...
I20250626 19:58:23.437925 7261 heartbeater.cc:507] Master 127.4.46.188:38919 requested a full tablet report, sending...
I20250626 19:58:23.438745 7262 heartbeater.cc:507] Master 127.4.46.189:36417 requested a full tablet report, sending...
I20250626 19:58:23.439472 7263 heartbeater.cc:344] Connected to a master server at 127.4.46.190:39311
I20250626 19:58:23.439793 7263 heartbeater.cc:461] Registering TS with master...
I20250626 19:58:23.440565 7263 heartbeater.cc:507] Master 127.4.46.190:39311 requested a full tablet report, sending...
I20250626 19:58:23.440989 6870 ts_manager.cc:194] Registered new tserver with Master: 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:23.441380 6792 ts_manager.cc:194] Registered new tserver with Master: 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:23.442655 6717 ts_manager.cc:194] Registered new tserver with Master: 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:23.444380 6717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.46.130:45539
I20250626 19:58:23.716022 7277 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:23.716598 7277 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:23.718408 7277 leader_election.cc:290] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
I20250626 19:58:23.719314 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:23.719468 6880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6610553ba0df459085f7373935774377" is_pre_election: true
I20250626 19:58:23.721040 6779 leader_election.cc:304] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ae38ec946e14198ae4adc8dde9d66e8; no voters: 188d02ec81e645229acdc5eed98c9f9f, 6610553ba0df459085f7373935774377
I20250626 19:58:23.721709 7277 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 19:58:23.722278 7269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 19:58:23.722887 7269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 19:58:23.723474 7269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250626 19:58:23.728994 7278 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:23.729341 7278 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:23.731171 7278 leader_election.cc:290] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417)
I20250626 19:58:23.731837 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:23.732254 6801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" is_pre_election: true
I20250626 19:58:23.735047 6856 leader_election.cc:304] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6610553ba0df459085f7373935774377; no voters: 188d02ec81e645229acdc5eed98c9f9f, 8ae38ec946e14198ae4adc8dde9d66e8
I20250626 19:58:23.735744 7278 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 19:58:23.757879 7269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 19:58:23.758925 7269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.46.131
I20250626 19:58:23.795877 7269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.46.148:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.46.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.46.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.46.131
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6785
TSAN enabled
I20250626 19:58:23.797297 7269 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 19:58:23.799006 7269 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 19:58:23.815532 7280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:58:23.947528 7263 heartbeater.cc:499] Master 127.4.46.190:39311 was elected leader, sending a full tablet report...
I20250626 19:58:24.325696 7286 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:24.326408 7286 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:24.328339 7286 leader_election.cc:290] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417)
I20250626 19:58:24.328959 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:24.331452 6801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" is_pre_election: true
I20250626 19:58:24.332955 6856 leader_election.cc:304] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6610553ba0df459085f7373935774377; no voters: 188d02ec81e645229acdc5eed98c9f9f, 8ae38ec946e14198ae4adc8dde9d66e8
I20250626 19:58:24.333765 7286 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:58:24.665555 7287 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:24.665970 7287 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:24.667654 7287 leader_election.cc:290] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
I20250626 19:58:24.668573 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:24.668735 6880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6610553ba0df459085f7373935774377" is_pre_election: true
I20250626 19:58:24.670255 6779 leader_election.cc:304] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ae38ec946e14198ae4adc8dde9d66e8; no voters: 188d02ec81e645229acdc5eed98c9f9f, 6610553ba0df459085f7373935774377
I20250626 19:58:24.670812 7287 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 19:58:23.817121 7281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:58:23.818881 7269 server_base.cc:1048] running on GCE node
W20250626 19:58:23.818173 7283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:58:25.014583 7269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 19:58:25.017077 7269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 19:58:25.018546 7269 hybrid_clock.cc:648] HybridClock initialized: now 1750967905018473 us; error 85 us; skew 500 ppm
I20250626 19:58:25.019716 7269 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 19:58:25.030982 7269 webserver.cc:469] Webserver started at http://127.4.46.131:33065/ using document root <none> and password file <none>
I20250626 19:58:25.031956 7269 fs_manager.cc:362] Metadata directory not provided
I20250626 19:58:25.032156 7269 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 19:58:25.032629 7269 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 19:58:25.037297 7269 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/data/instance:
uuid: "297d2c913667408bae82668a557e06c7"
format_stamp: "Formatted at 2025-06-26 19:58:25 on dist-test-slave-48dm"
I20250626 19:58:25.038424 7269 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/instance:
uuid: "297d2c913667408bae82668a557e06c7"
format_stamp: "Formatted at 2025-06-26 19:58:25 on dist-test-slave-48dm"
I20250626 19:58:25.046660 7269 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20250626 19:58:25.054198 7292 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 19:58:25.055475 7269 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 19:58:25.055816 7269 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal
uuid: "297d2c913667408bae82668a557e06c7"
format_stamp: "Formatted at 2025-06-26 19:58:25 on dist-test-slave-48dm"
I20250626 19:58:25.056154 7269 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 19:58:25.110736 7269 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 19:58:25.112577 7269 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 19:58:25.113034 7269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 19:58:25.115613 7269 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 19:58:25.119957 7269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 19:58:25.120178 7269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 19:58:25.120460 7269 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 19:58:25.120652 7269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 19:58:25.179850 7333 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:25.180291 7333 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:25.182834 7333 leader_election.cc:290] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
I20250626 19:58:25.183147 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:25.184226 6880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6610553ba0df459085f7373935774377" is_pre_election: true
I20250626 19:58:25.185655 6779 leader_election.cc:304] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ae38ec946e14198ae4adc8dde9d66e8; no voters: 188d02ec81e645229acdc5eed98c9f9f, 6610553ba0df459085f7373935774377
I20250626 19:58:25.186640 7333 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:58:25.273250 7269 rpc_server.cc:307] RPC server started. Bound to: 127.4.46.131:33703
I20250626 19:58:25.273386 7405 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.46.131:33703 every 8 connection(s)
I20250626 19:58:25.275921 7269 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/data/info.pb
I20250626 19:58:25.281486 4282 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu as pid 7269
I20250626 19:58:25.282466 4282 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/instance
W20250626 19:58:25.318601 6749 debug-util.cc:398] Leaking SignalData structure 0x7b080006f240 after lost signal to thread 6686
I20250626 19:58:25.335462 7408 heartbeater.cc:344] Connected to a master server at 127.4.46.190:39311
I20250626 19:58:25.337500 7408 heartbeater.cc:461] Registering TS with master...
I20250626 19:58:25.338871 7408 heartbeater.cc:507] Master 127.4.46.190:39311 requested a full tablet report, sending...
I20250626 19:58:25.341830 6717 ts_manager.cc:194] Registered new tserver with Master: 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:25.342324 7407 heartbeater.cc:344] Connected to a master server at 127.4.46.189:36417
I20250626 19:58:25.342691 7407 heartbeater.cc:461] Registering TS with master...
I20250626 19:58:25.343673 7407 heartbeater.cc:507] Master 127.4.46.189:36417 requested a full tablet report, sending...
I20250626 19:58:25.343698 6717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.46.131:42365
I20250626 19:58:25.344250 7406 heartbeater.cc:344] Connected to a master server at 127.4.46.188:38919
I20250626 19:58:25.344578 7406 heartbeater.cc:461] Registering TS with master...
I20250626 19:58:25.345427 7406 heartbeater.cc:507] Master 127.4.46.188:38919 requested a full tablet report, sending...
I20250626 19:58:25.347189 6792 ts_manager.cc:194] Registered new tserver with Master: 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:25.348065 6870 ts_manager.cc:194] Registered new tserver with Master: 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:25.381565 4282 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 19:58:25.444581 6717 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_d68d3f0aa49c4afb89ce351c9baf0a2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:25.447198 6718 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_d0b4adacbfa84fbc8979bd9fe8d5bca7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:25.451932 6716 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_35326ce5948a4f5ab528b8243ecb7479"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:25.453636 6715 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_e1291c46a74d4983aac77cc888bd7962"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:58:25.479254 6717 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:58:25.482672 6716 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_35326ce5948a4f5ab528b8243ecb7479 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:58:25.485193 6718 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:25.552582 4282 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu with pid 6754
W20250626 19:58:25.748034 6704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20250626 19:58:25.769635 6704 consensus_peers.cc:487] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f -> Peer 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417): Couldn't send request to peer 8ae38ec946e14198ae4adc8dde9d66e8. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 19:58:25.848801 7408 heartbeater.cc:499] Master 127.4.46.190:39311 was elected leader, sending a full tablet report...
W20250626 19:58:25.864359 7407 heartbeater.cc:646] Failed to heartbeat to 127.4.46.189:36417 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 19:58:25.983093 7262 heartbeater.cc:646] Failed to heartbeat to 127.4.46.189:36417 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 19:58:26.048281 7119 heartbeater.cc:646] Failed to heartbeat to 127.4.46.189:36417 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
I20250626 19:58:26.104506 6717 catalog_manager.cc:2149] created HMS catalog entry for table [id=ee116b84a24e49228c7496e95eaf11bb]
I20250626 19:58:26.155298 6716 catalog_manager.cc:2149] created HMS catalog entry for table [id=eff2225412944029aed88f6c87744fc8]
I20250626 19:58:26.193581 4282 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
/tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.189:36417
--webserver_interface=127.4.46.189
--webserver_port=37313
--builtin_ntp_servers=127.4.46.148:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919
--hive_metastore_uris=thrift://127.0.0.1:33633
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250626 19:58:26.215619 6973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "188d02ec81e645229acdc5eed98c9f9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:26.216902 6973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: This master's current role is: LEADER
I20250626 19:58:26.250525 6718 catalog_manager.cc:2149] created HMS catalog entry for table [id=8b447518f0d84428b97e7ffee1a90a2c]
I20250626 19:58:26.250867 6716 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_35326ce5948a4f5ab528b8243ecb7479" } new_table_name: "default.Table_a5221df6a6b54bcdafa382fca51b613d" modify_external_catalogs: true
I20250626 19:58:26.252233 6717 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_d68d3f0aa49c4afb89ce351c9baf0a2d" } new_table_name: "default.Table_252448e828024438ba317b8277b66523" modify_external_catalogs: true
W20250626 19:58:26.254434 6715 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e1291c46a74d4983aac77cc888bd7962 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:26.289240 6718 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_d0b4adacbfa84fbc8979bd9fe8d5bca7" } modify_external_catalogs: true
I20250626 19:58:26.291007 6718 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_d0b4adacbfa84fbc8979bd9fe8d5bca7" } modify_external_catalogs: true
I20250626 19:58:26.341236 7195 tablet_service.cc:1468] Processing CreateTablet for tablet ae04997b77924e8ba479c5c34ac822d7 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:26.342703 7195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae04997b77924e8ba479c5c34ac822d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:26.336566 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 5773ef416eb2472d99b27dd0d8fb0ca8 (DEFAULT_TABLE table=default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:26.352135 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5773ef416eb2472d99b27dd0d8fb0ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:26.409088 7341 tablet_service.cc:1468] Processing CreateTablet for tablet 5773ef416eb2472d99b27dd0d8fb0ca8 (DEFAULT_TABLE table=default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:26.411134 7341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5773ef416eb2472d99b27dd0d8fb0ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:26.416371 7340 tablet_service.cc:1468] Processing CreateTablet for tablet ae04997b77924e8ba479c5c34ac822d7 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:26.417558 7340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae04997b77924e8ba479c5c34ac822d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:26.422595 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 734d9d3331b44fc49e1f64b8caf1e4d8 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:26.423843 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 734d9d3331b44fc49e1f64b8caf1e4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:26.433044 6715 catalog_manager.cc:2149] created HMS catalog entry for table [id=52b892ca731141a0870ab2cf2864f46f]
I20250626 19:58:26.451521 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 734d9d3331b44fc49e1f64b8caf1e4d8 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250626 19:58:26.453558 6705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.188:38919, user_credentials={real_user=slave}} blocked reactor thread for 48163.3us
I20250626 19:58:26.461143 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 734d9d3331b44fc49e1f64b8caf1e4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:26.501428 6714 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_f9f594403d34451f8f2fa6d607d33a2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:26.503011 7053 tablet_service.cc:1468] Processing CreateTablet for tablet 5773ef416eb2472d99b27dd0d8fb0ca8 (DEFAULT_TABLE table=default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:26.506639 7053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5773ef416eb2472d99b27dd0d8fb0ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:26.505079 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 734d9d3331b44fc49e1f64b8caf1e4d8 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:26.504441 7052 tablet_service.cc:1468] Processing CreateTablet for tablet ae04997b77924e8ba479c5c34ac822d7 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:26.508903 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 734d9d3331b44fc49e1f64b8caf1e4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:26.509599 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae04997b77924e8ba479c5c34ac822d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:26.512327 6715 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 4) took 1059 ms. Trace:
I20250626 19:58:26.512743 6715 rpcz_store.cc:276] 0626 19:58:25.452923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:25.453094 (+ 171us) service_pool.cc:225] Handling call
0626 19:58:26.254983 (+801889us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:26.256821 (+ 1838us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:26.432995 (+176174us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:26.511625 (+ 78630us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:26.511960 (+ 335us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:26.512249 (+ 289us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:26.435091 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:26.435580 (+ 489us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:26.435611 (+ 31us) write_op.cc:435] Acquired schema lock
0626 19:58:26.435625 (+ 14us) tablet.cc:660] Decoding operations
0626 19:58:26.435964 (+ 339us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:26.436011 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:26.436041 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:26.436277 (+ 236us) tablet.cc:699] Row locks acquired
0626 19:58:26.436296 (+ 19us) write_op.cc:260] PREPARE: finished
0626 19:58:26.436421 (+ 125us) write_op.cc:270] Start()
0626 19:58:26.436525 (+ 104us) write_op.cc:276] Timestamp: P: 1750967906436410 usec, L: 0
0626 19:58:26.436547 (+ 22us) op_driver.cc:348] REPLICATION: starting
0626 19:58:26.437132 (+ 585us) log.cc:844] Serialized 432 byte log entry
0626 19:58:26.509308 (+ 72176us) op_driver.cc:464] REPLICATION: finished
0626 19:58:26.509759 (+ 451us) write_op.cc:301] APPLY: starting
0626 19:58:26.509821 (+ 62us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:26.509929 (+ 108us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:26.510016 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:26.510391 (+ 375us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:26.510565 (+ 174us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:26.510613 (+ 48us) write_op.cc:312] APPLY: finished
0626 19:58:26.510841 (+ 228us) log.cc:844] Serialized 28 byte log entry
0626 19:58:26.510989 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:26.511061 (+ 72us) write_op.cc:454] Released schema lock
0626 19:58:26.511095 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0626 19:58:26.511325 (+ 230us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:26.511351 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":5764,"Hive Metastore.run_cpu_time_us":684,"Hive Metastore.run_wall_time_us":169502,"mutex_wait_us":14630,"spinlock_wait_cycles":2432,"child_traces":[["op",{"apply.queue_time_us":301,"apply.run_cpu_time_us":1517,"apply.run_wall_time_us":1523,"mutex_wait_us":15048,"num_ops":2,"prepare.queue_time_us":125,"prepare.run_cpu_time_us":2413,"prepare.run_wall_time_us":17442,"raft.queue_time_us":15284,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":13,"replication_time_us":72639}]]}
I20250626 19:58:26.550573 6713 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_64bb792c0fa74fab8347bf7bda0dedbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:26.554418 6715 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_5283819977d9487a9d1aaae326249a46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:26.572369 6712 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_e1291c46a74d4983aac77cc888bd7962" } modify_external_catalogs: true
I20250626 19:58:26.573185 6712 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_e1291c46a74d4983aac77cc888bd7962" } modify_external_catalogs: true
I20250626 19:58:26.579941 7050 tablet_service.cc:1468] Processing CreateTablet for tablet ec11cee9ebe94835923f8ecb1e856bfe (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:26.581091 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec11cee9ebe94835923f8ecb1e856bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:26.581525 7193 tablet_service.cc:1468] Processing CreateTablet for tablet ec11cee9ebe94835923f8ecb1e856bfe (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:26.581665 7338 tablet_service.cc:1468] Processing CreateTablet for tablet ec11cee9ebe94835923f8ecb1e856bfe (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:26.582790 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec11cee9ebe94835923f8ecb1e856bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:26.582837 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec11cee9ebe94835923f8ecb1e856bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:26.612108 7464 tablet_bootstrap.cc:492] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:26.624265 7467 tablet_bootstrap.cc:492] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:26.636786 7464 tablet_bootstrap.cc:654] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:26.639417 7466 tablet_bootstrap.cc:492] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:26.640142 7464 log.cc:826] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a: Log is configured to *not* fsync() on all Append() calls
I20250626 19:58:26.642465 7467 tablet_bootstrap.cc:654] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:26.645913 7467 log.cc:826] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7: Log is configured to *not* fsync() on all Append() calls
I20250626 19:58:26.650252 7466 tablet_bootstrap.cc:654] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:26.651180 7464 tablet_bootstrap.cc:492] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:26.651893 7464 ts_tablet_manager.cc:1397] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.007s
I20250626 19:58:26.657351 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet ae04997b77924e8ba479c5c34ac822d7 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:26.671226 6710 master_service.cc:948] ReplaceTablet: received request to replace tablet ae04997b77924e8ba479c5c34ac822d7 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:26.679857 7466 log.cc:826] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca: Log is configured to *not* fsync() on all Append() calls
I20250626 19:58:26.680395 7467 tablet_bootstrap.cc:492] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:26.680934 7467 ts_tablet_manager.cc:1397] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.060s user 0.013s sys 0.000s
I20250626 19:58:26.691489 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet ae04997b77924e8ba479c5c34ac822d7
I20250626 19:58:26.693176 6711 catalog_manager.cc:6652] ReplaceTablet: tablet ae04997b77924e8ba479c5c34ac822d7 deleted and replaced by tablet 09c15f0a651046a8922074134769806e
I20250626 19:58:26.700197 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet ae04997b77924e8ba479c5c34ac822d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 09c15f0a651046a8922074134769806e) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:26.702930 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet ae04997b77924e8ba479c5c34ac822d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 09c15f0a651046a8922074134769806e) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:26.704985 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet ae04997b77924e8ba479c5c34ac822d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 09c15f0a651046a8922074134769806e) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:26.705845 7466 tablet_bootstrap.cc:492] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:26.706614 7466 ts_tablet_manager.cc:1397] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.009s
W20250626 19:58:26.718644 6702 catalog_manager.cc:4908] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet ae04997b77924e8ba479c5c34ac822d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae04997b77924e8ba479c5c34ac822d7 already in progress: creating tablet
I20250626 19:58:26.730283 7467 raft_consensus.cc:357] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:26.736310 7467 raft_consensus.cc:383] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:26.736864 7467 raft_consensus.cc:738] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:26.738649 7467 consensus_queue.cc:260] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:26.740724 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 09c15f0a651046a8922074134769806e (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:26.741820 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c15f0a651046a8922074134769806e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:58:26.744475 6703 catalog_manager.cc:4908] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet ae04997b77924e8ba479c5c34ac822d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae04997b77924e8ba479c5c34ac822d7 already in progress: creating tablet
I20250626 19:58:26.744685 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 09c15f0a651046a8922074134769806e (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:26.745729 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c15f0a651046a8922074134769806e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:26.755945 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 09c15f0a651046a8922074134769806e (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:26.757112 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c15f0a651046a8922074134769806e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:26.757010 6710 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_a050ca8b9a9043ee87ca530228790f54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:58:26.759919 6705 catalog_manager.cc:4908] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet ae04997b77924e8ba479c5c34ac822d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae04997b77924e8ba479c5c34ac822d7 already in progress: creating tablet
I20250626 19:58:26.789484 7464 raft_consensus.cc:357] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:26.795830 7464 raft_consensus.cc:383] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:26.796205 7464 raft_consensus.cc:738] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:26.800272 7464 consensus_queue.cc:260] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:26.821167 7467 ts_tablet_manager.cc:1428] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.140s user 0.042s sys 0.028s
I20250626 19:58:26.837838 7466 raft_consensus.cc:357] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:26.840950 7467 tablet_bootstrap.cc:492] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:26.848551 7464 ts_tablet_manager.cc:1428] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.196s user 0.032s sys 0.000s
I20250626 19:58:26.853327 7466 raft_consensus.cc:383] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:26.853803 7466 raft_consensus.cc:738] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:26.854851 7466 consensus_queue.cc:260] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:26.862804 7467 tablet_bootstrap.cc:654] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:26.869683 7464 tablet_bootstrap.cc:492] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:26.873950 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet ae04997b77924e8ba479c5c34ac822d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 09c15f0a651046a8922074134769806e) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:26.880746 7466 ts_tablet_manager.cc:1428] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.174s user 0.026s sys 0.007s
I20250626 19:58:26.881742 7466 tablet_bootstrap.cc:492] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:26.883221 7467 tablet_bootstrap.cc:492] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:26.883683 7467 ts_tablet_manager.cc:1397] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.004s
I20250626 19:58:26.886324 7467 raft_consensus.cc:357] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:26.887157 7467 raft_consensus.cc:383] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:26.887523 7467 raft_consensus.cc:738] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:26.888303 7467 consensus_queue.cc:260] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:26.900044 7466 tablet_bootstrap.cc:654] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:26.900574 7467 ts_tablet_manager.cc:1428] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.017s user 0.003s sys 0.002s
I20250626 19:58:26.909466 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet 734d9d3331b44fc49e1f64b8caf1e4d8 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:26.909600 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet ae04997b77924e8ba479c5c34ac822d7 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:26.919349 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet ae04997b77924e8ba479c5c34ac822d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 09c15f0a651046a8922074134769806e) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:26.920356 7471 tablet_replica.cc:331] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:26.921191 7471 raft_consensus.cc:2241] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:26.924696 7471 raft_consensus.cc:2270] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:26.927495 7471 ts_tablet_manager.cc:1905] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:26.940671 7464 tablet_bootstrap.cc:654] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:26.920725 7467 tablet_bootstrap.cc:492] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:26.940986 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 734d9d3331b44fc49e1f64b8caf1e4d8
I20250626 19:58:26.941546 7473 tablet_replica.cc:331] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:26.942404 7473 raft_consensus.cc:2241] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:26.943071 7473 raft_consensus.cc:2270] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:26.944041 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 734d9d3331b44fc49e1f64b8caf1e4d8 with delete_type TABLET_DATA_DELETED (replaced by tablet 74b320a2d10a490f90c0ef49c802e4fa) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:26.944600 7466 tablet_bootstrap.cc:492] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:26.945026 7466 ts_tablet_manager.cc:1397] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.063s user 0.007s sys 0.004s
I20250626 19:58:26.947933 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 734d9d3331b44fc49e1f64b8caf1e4d8 with delete_type TABLET_DATA_DELETED (replaced by tablet 74b320a2d10a490f90c0ef49c802e4fa) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:26.948211 6711 catalog_manager.cc:6652] ReplaceTablet: tablet 734d9d3331b44fc49e1f64b8caf1e4d8 deleted and replaced by tablet 74b320a2d10a490f90c0ef49c802e4fa
W20250626 19:58:26.949584 6703 catalog_manager.cc:4908] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 734d9d3331b44fc49e1f64b8caf1e4d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 734d9d3331b44fc49e1f64b8caf1e4d8 already in progress: creating tablet
I20250626 19:58:26.950729 7473 ts_tablet_manager.cc:1905] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:26.947926 7466 raft_consensus.cc:357] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:26.951606 7466 raft_consensus.cc:383] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:26.951930 7466 raft_consensus.cc:738] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:26.946658 7467 tablet_bootstrap.cc:654] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:26.943501 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 734d9d3331b44fc49e1f64b8caf1e4d8 with delete_type TABLET_DATA_DELETED (replaced by tablet 74b320a2d10a490f90c0ef49c802e4fa) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:26.957907 7466 consensus_queue.cc:260] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:26.965420 7466 ts_tablet_manager.cc:1428] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
W20250626 19:58:26.967063 7121 tablet.cc:2378] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 19:58:26.973245 7466 tablet_bootstrap.cc:492] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:26.982729 7473 ts_tablet_manager.cc:1918] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:26.990999 7473 log.cc:1199] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/ae04997b77924e8ba479c5c34ac822d7
I20250626 19:58:26.992511 7473 ts_tablet_manager.cc:1939] T ae04997b77924e8ba479c5c34ac822d7 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:26.979666 7466 tablet_bootstrap.cc:654] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:26.996450 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet ae04997b77924e8ba479c5c34ac822d7 (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:27.002460 7471 ts_tablet_manager.cc:1918] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:27.004541 7467 tablet_bootstrap.cc:492] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:27.005039 7467 ts_tablet_manager.cc:1397] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.009s
I20250626 19:58:27.008087 7467 raft_consensus.cc:357] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:27.009829 6705 catalog_manager.cc:4908] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 734d9d3331b44fc49e1f64b8caf1e4d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 734d9d3331b44fc49e1f64b8caf1e4d8 already in progress: creating tablet
I20250626 19:58:27.010711 7467 raft_consensus.cc:383] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.011046 7471 log.cc:1199] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/ae04997b77924e8ba479c5c34ac822d7
I20250626 19:58:27.011066 7467 raft_consensus.cc:738] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.012017 7471 ts_tablet_manager.cc:1939] T ae04997b77924e8ba479c5c34ac822d7 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:27.015003 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet ae04997b77924e8ba479c5c34ac822d7 (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:27.006556 7464 tablet_bootstrap.cc:492] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:27.015870 7464 ts_tablet_manager.cc:1397] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.146s user 0.011s sys 0.000s
I20250626 19:58:27.017477 7466 tablet_bootstrap.cc:492] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:27.017943 7466 ts_tablet_manager.cc:1397] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.000s
I20250626 19:58:27.017601 7467 consensus_queue.cc:260] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.019263 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 74b320a2d10a490f90c0ef49c802e4fa (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:27.019760 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 74b320a2d10a490f90c0ef49c802e4fa (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:27.020262 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b320a2d10a490f90c0ef49c802e4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:27.020761 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b320a2d10a490f90c0ef49c802e4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:27.026793 7467 ts_tablet_manager.cc:1428] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.021s user 0.002s sys 0.003s
I20250626 19:58:27.030253 7467 tablet_bootstrap.cc:492] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:27.029843 7466 raft_consensus.cc:357] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.030539 7466 raft_consensus.cc:383] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.035486 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 74b320a2d10a490f90c0ef49c802e4fa (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:27.036516 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b320a2d10a490f90c0ef49c802e4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:27.048434 7467 tablet_bootstrap.cc:654] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:27.053550 7466 raft_consensus.cc:738] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.054319 7466 consensus_queue.cc:260] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.060350 7466 ts_tablet_manager.cc:1428] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.042s user 0.005s sys 0.000s
W20250626 19:58:27.061954 7413 tablet.cc:2378] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 19:58:27.064397 7466 tablet_bootstrap.cc:492] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:27.088377 7466 tablet_bootstrap.cc:654] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:27.091341 6709 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
I20250626 19:58:27.096119 7464 raft_consensus.cc:357] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.112036 7466 tablet_bootstrap.cc:492] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:27.112262 7464 raft_consensus.cc:383] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.112663 7464 raft_consensus.cc:738] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.113451 7464 consensus_queue.cc:260] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:27.115276 6702 catalog_manager.cc:4908] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet 734d9d3331b44fc49e1f64b8caf1e4d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 734d9d3331b44fc49e1f64b8caf1e4d8 already in progress: creating tablet
I20250626 19:58:27.119524 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet 5773ef416eb2472d99b27dd0d8fb0ca8 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:27.122669 7466 ts_tablet_manager.cc:1397] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.001s
I20250626 19:58:27.133459 7466 raft_consensus.cc:357] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.138085 7466 raft_consensus.cc:383] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.138469 7466 raft_consensus.cc:738] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.139187 7466 consensus_queue.cc:260] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.142794 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 5773ef416eb2472d99b27dd0d8fb0ca8
I20250626 19:58:27.145022 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 5773ef416eb2472d99b27dd0d8fb0ca8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7dcdf6675ca24b7e93c7d545f066efc2) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:27.146121 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 5773ef416eb2472d99b27dd0d8fb0ca8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7dcdf6675ca24b7e93c7d545f066efc2) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:27.146481 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 5773ef416eb2472d99b27dd0d8fb0ca8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7dcdf6675ca24b7e93c7d545f066efc2) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:27.146996 7472 tablet_replica.cc:331] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:27.147257 7473 tablet_replica.cc:331] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:27.147841 7472 raft_consensus.cc:2241] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:27.147948 7473 raft_consensus.cc:2241] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:27.148367 7472 raft_consensus.cc:2270] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:27.148404 7473 raft_consensus.cc:2270] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:27.150537 7473 ts_tablet_manager.cc:1905] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:27.151018 7472 ts_tablet_manager.cc:1905] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:27.153151 7464 ts_tablet_manager.cc:1428] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.060s user 0.005s sys 0.000s
W20250626 19:58:27.154479 6702 catalog_manager.cc:4908] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet 5773ef416eb2472d99b27dd0d8fb0ca8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5773ef416eb2472d99b27dd0d8fb0ca8 already in progress: creating tablet
I20250626 19:58:27.161818 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 09c15f0a651046a8922074134769806e from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:27.165524 6711 catalog_manager.cc:6652] ReplaceTablet: tablet 5773ef416eb2472d99b27dd0d8fb0ca8 deleted and replaced by tablet 7dcdf6675ca24b7e93c7d545f066efc2
I20250626 19:58:27.166841 7477 raft_consensus.cc:491] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:27.167383 7477 raft_consensus.cc:513] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.170395 7477 leader_election.cc:290] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:27.187631 7466 ts_tablet_manager.cc:1428] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.065s user 0.003s sys 0.003s
I20250626 19:58:27.192690 7473 ts_tablet_manager.cc:1918] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250626 19:58:27.202781 7268 tablet.cc:2378] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 19:58:27.206399 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 09c15f0a651046a8922074134769806e
I20250626 19:58:27.209867 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 09c15f0a651046a8922074134769806e deleted and replaced by tablet d17c208f1d4b46649145e1d0de155c5d
I20250626 19:58:27.209862 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 09c15f0a651046a8922074134769806e with delete_type TABLET_DATA_DELETED (replaced by tablet d17c208f1d4b46649145e1d0de155c5d) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:27.210958 7473 log.cc:1199] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/5773ef416eb2472d99b27dd0d8fb0ca8
I20250626 19:58:27.211758 7473 ts_tablet_manager.cc:1939] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:27.208158 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 09c15f0a651046a8922074134769806e with delete_type TABLET_DATA_DELETED (replaced by tablet d17c208f1d4b46649145e1d0de155c5d) from {username='slave'} at 127.0.0.1:54698
W20250626 19:58:27.212880 7453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20250626 19:58:27.213421 7466 tablet_bootstrap.cc:492] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
W20250626 19:58:27.217574 7453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 19:58:27.219004 7453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250626 19:58:27.237681 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 7dcdf6675ca24b7e93c7d545f066efc2 (DEFAULT_TABLE table=default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:27.238826 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dcdf6675ca24b7e93c7d545f066efc2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:58:27.239945 6711 catalog_manager.cc:5415] Ignoring report from unknown tablet ae04997b77924e8ba479c5c34ac822d7
I20250626 19:58:27.243590 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 7dcdf6675ca24b7e93c7d545f066efc2 (DEFAULT_TABLE table=default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:27.244666 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dcdf6675ca24b7e93c7d545f066efc2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:58:27.246996 6702 catalog_manager.cc:4908] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet 09c15f0a651046a8922074134769806e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09c15f0a651046a8922074134769806e already in progress: creating tablet
I20250626 19:58:27.251765 7464 tablet_bootstrap.cc:492] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
W20250626 19:58:27.260058 6709 catalog_manager.cc:5415] Ignoring report from unknown tablet ae04997b77924e8ba479c5c34ac822d7
I20250626 19:58:27.268639 7467 tablet_bootstrap.cc:492] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:27.269496 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet 5773ef416eb2472d99b27dd0d8fb0ca8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7dcdf6675ca24b7e93c7d545f066efc2) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:27.269174 7467 ts_tablet_manager.cc:1397] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.239s user 0.051s sys 0.060s
I20250626 19:58:27.270444 7471 tablet_replica.cc:331] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:27.270838 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 5773ef416eb2472d99b27dd0d8fb0ca8 (table default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:27.272338 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 09c15f0a651046a8922074134769806e with delete_type TABLET_DATA_DELETED (replaced by tablet d17c208f1d4b46649145e1d0de155c5d) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:27.273010 7467 raft_consensus.cc:357] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.274183 7471 raft_consensus.cc:2241] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:27.274681 7471 raft_consensus.cc:2270] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250626 19:58:27.274711 6705 catalog_manager.cc:4908] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 09c15f0a651046a8922074134769806e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09c15f0a651046a8922074134769806e already in progress: creating tablet
I20250626 19:58:27.273940 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 7dcdf6675ca24b7e93c7d545f066efc2 (DEFAULT_TABLE table=default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:27.275897 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dcdf6675ca24b7e93c7d545f066efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:27.286868 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "734d9d3331b44fc49e1f64b8caf1e4d8" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:27.288523 7471 ts_tablet_manager.cc:1905] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:58:27.288616 7006 leader_election.cc:343] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:27.288902 7467 raft_consensus.cc:383] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.289278 7467 raft_consensus.cc:738] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.295078 7466 tablet_bootstrap.cc:654] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:27.294790 7467 consensus_queue.cc:260] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.302407 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "734d9d3331b44fc49e1f64b8caf1e4d8" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:27.303239 7361 raft_consensus.cc:2466] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:27.306094 7472 ts_tablet_manager.cc:1918] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:27.308599 7467 ts_tablet_manager.cc:1428] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
I20250626 19:58:27.316202 7467 tablet_bootstrap.cc:492] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:27.316430 7009 leader_election.cc:304] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters: 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:27.317349 7477 raft_consensus.cc:2802] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:27.317695 7477 raft_consensus.cc:491] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:27.317999 7477 raft_consensus.cc:3058] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:27.319584 6711 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
W20250626 19:58:27.325238 6709 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
I20250626 19:58:27.325587 7467 tablet_bootstrap.cc:654] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:27.326525 7477 raft_consensus.cc:513] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.329499 7052 tablet_service.cc:1468] Processing CreateTablet for tablet d17c208f1d4b46649145e1d0de155c5d (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:27.330099 7464 tablet_bootstrap.cc:654] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:27.330679 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d17c208f1d4b46649145e1d0de155c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:27.345522 7477 leader_election.cc:290] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Requested vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:27.345625 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "734d9d3331b44fc49e1f64b8caf1e4d8" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:27.343518 7338 tablet_service.cc:1468] Processing CreateTablet for tablet d17c208f1d4b46649145e1d0de155c5d (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250626 19:58:27.346660 7006 leader_election.cc:343] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:27.346684 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "734d9d3331b44fc49e1f64b8caf1e4d8" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:27.306631 7472 log.cc:1199] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/5773ef416eb2472d99b27dd0d8fb0ca8
I20250626 19:58:27.347299 7464 tablet_bootstrap.cc:492] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:27.347287 7361 raft_consensus.cc:3058] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:27.347534 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d17c208f1d4b46649145e1d0de155c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:27.347725 7464 ts_tablet_manager.cc:1397] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.096s user 0.011s sys 0.000s
I20250626 19:58:27.348323 7472 ts_tablet_manager.cc:1939] T 5773ef416eb2472d99b27dd0d8fb0ca8 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:27.350430 7464 raft_consensus.cc:357] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.360205 7361 raft_consensus.cc:2466] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:27.361346 7009 leader_election.cc:304] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters: 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:27.362879 7464 raft_consensus.cc:383] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.363241 7464 raft_consensus.cc:738] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.356484 7471 ts_tablet_manager.cc:1918] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:27.364164 7471 log.cc:1199] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/5773ef416eb2472d99b27dd0d8fb0ca8
I20250626 19:58:27.363981 7464 consensus_queue.cc:260] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.364861 7471 ts_tablet_manager.cc:1939] T 5773ef416eb2472d99b27dd0d8fb0ca8 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:27.368393 7464 ts_tablet_manager.cc:1428] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250626 19:58:27.369637 7464 tablet_bootstrap.cc:492] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:27.376869 7464 tablet_bootstrap.cc:654] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:27.382790 7477 raft_consensus.cc:2802] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:27.385517 7194 tablet_service.cc:1468] Processing CreateTablet for tablet d17c208f1d4b46649145e1d0de155c5d (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:27.386866 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d17c208f1d4b46649145e1d0de155c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:27.388883 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 5773ef416eb2472d99b27dd0d8fb0ca8 (table default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:27.391881 7466 tablet_bootstrap.cc:492] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:27.392414 7466 ts_tablet_manager.cc:1397] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.179s user 0.012s sys 0.000s
I20250626 19:58:27.398010 7464 tablet_bootstrap.cc:492] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:27.401417 7464 ts_tablet_manager.cc:1397] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.000s
I20250626 19:58:27.402449 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 5773ef416eb2472d99b27dd0d8fb0ca8 (table default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
W20250626 19:58:27.403151 6703 catalog_manager.cc:4908] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 09c15f0a651046a8922074134769806e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09c15f0a651046a8922074134769806e already in progress: creating tablet
I20250626 19:58:27.404044 7464 raft_consensus.cc:357] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.404754 7467 tablet_bootstrap.cc:492] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:27.404713 7464 raft_consensus.cc:383] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.404986 7464 raft_consensus.cc:738] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.405218 7467 ts_tablet_manager.cc:1397] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.089s user 0.006s sys 0.004s
I20250626 19:58:27.405644 7464 consensus_queue.cc:260] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.407934 7467 raft_consensus.cc:357] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.408614 7467 raft_consensus.cc:383] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.408874 7467 raft_consensus.cc:738] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.409554 7464 ts_tablet_manager.cc:1428] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250626 19:58:27.410425 7464 tablet_bootstrap.cc:492] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
W20250626 19:58:27.417610 6711 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
I20250626 19:58:27.418243 7467 consensus_queue.cc:260] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.419307 7474 raft_consensus.cc:491] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:27.419740 7474 raft_consensus.cc:513] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.420456 7477 raft_consensus.cc:695] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Becoming Leader. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Running, Role: LEADER
I20250626 19:58:27.421475 7477 consensus_queue.cc:237] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.430241 7474 leader_election.cc:290] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
W20250626 19:58:27.432207 6711 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
I20250626 19:58:27.432438 7467 ts_tablet_manager.cc:1428] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20250626 19:58:27.433346 7467 tablet_bootstrap.cc:492] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:27.433358 7464 tablet_bootstrap.cc:654] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:27.435837 7498 raft_consensus.cc:491] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:27.463320 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 09c15f0a651046a8922074134769806e with delete_type TABLET_DATA_DELETED (replaced by tablet d17c208f1d4b46649145e1d0de155c5d) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:27.465845 7473 tablet_replica.cc:331] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:27.466827 7473 raft_consensus.cc:2241] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:27.467430 7473 raft_consensus.cc:2270] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:27.469696 7473 ts_tablet_manager.cc:1905] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:27.485270 7467 tablet_bootstrap.cc:654] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:27.485380 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09c15f0a651046a8922074134769806e" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:27.457806 7498 raft_consensus.cc:513] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.489338 7464 tablet_bootstrap.cc:492] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:27.489764 7464 ts_tablet_manager.cc:1397] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.080s user 0.026s sys 0.005s
I20250626 19:58:27.491209 7498 leader_election.cc:290] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:27.491492 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec11cee9ebe94835923f8ecb1e856bfe" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:27.492193 7216 raft_consensus.cc:2466] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:27.492265 7464 raft_consensus.cc:357] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.492954 7464 raft_consensus.cc:383] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.493340 7006 leader_election.cc:304] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:27.496644 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09c15f0a651046a8922074134769806e" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
W20250626 19:58:27.496986 7453 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250626 19:58:27.497337 7453 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
I20250626 19:58:27.497435 7464 raft_consensus.cc:738] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
W20250626 19:58:27.497805 7453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 19:58:27.498196 7453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 19:58:27.498577 7453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20250626 19:58:27.498190 7464 consensus_queue.cc:260] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:27.498998 7453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 19:58:27.395680 7466 raft_consensus.cc:357] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.499758 7466 raft_consensus.cc:383] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.500114 7466 raft_consensus.cc:738] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.500818 7466 consensus_queue.cc:260] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.501761 7073 raft_consensus.cc:2466] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:27.506385 7464 ts_tablet_manager.cc:1428] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.016s user 0.002s sys 0.004s
I20250626 19:58:27.508370 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 09c15f0a651046a8922074134769806e with delete_type TABLET_DATA_DELETED (replaced by tablet d17c208f1d4b46649145e1d0de155c5d) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:27.512887 7471 tablet_replica.cc:331] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:27.513607 7471 raft_consensus.cc:2241] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:27.514058 7471 raft_consensus.cc:2270] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:27.515605 7471 ts_tablet_manager.cc:1905] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:27.523358 7498 raft_consensus.cc:2802] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:27.523782 7498 raft_consensus.cc:491] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:27.524178 7498 raft_consensus.cc:3058] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:27.539192 7464 tablet_bootstrap.cc:492] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:27.542388 7466 ts_tablet_manager.cc:1428] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.150s user 0.004s sys 0.004s
I20250626 19:58:27.543437 7466 tablet_bootstrap.cc:492] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:27.547432 7464 tablet_bootstrap.cc:654] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:27.549417 6942 hms_client.cc:336] Time spent get HMS table: real 1.078s user 0.000s sys 0.000s
W20250626 19:58:27.487067 7293 leader_election.cc:343] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:58:27.553789 7293 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:40373, user_credentials={real_user=slave}} blocked reactor thread for 66936.6us
I20250626 19:58:27.556715 7294 leader_election.cc:304] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters: 315d304854c541bf8b5e6b1eea712d4a
W20250626 19:58:27.557559 7294 raft_consensus.cc:2684] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250626 19:58:27.558163 7294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:33129, user_credentials={real_user=slave}} blocked reactor thread for 52640.7us
I20250626 19:58:27.559042 7471 ts_tablet_manager.cc:1918] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:27.559494 7471 log.cc:1199] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/09c15f0a651046a8922074134769806e
I20250626 19:58:27.559455 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec11cee9ebe94835923f8ecb1e856bfe" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:27.559995 7361 raft_consensus.cc:2466] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:27.560242 7471 ts_tablet_manager.cc:1939] T 09c15f0a651046a8922074134769806e P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:27.563028 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 09c15f0a651046a8922074134769806e (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:27.565963 7464 tablet_bootstrap.cc:492] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:27.566398 7464 ts_tablet_manager.cc:1397] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.004s
I20250626 19:58:27.568926 7464 raft_consensus.cc:357] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.571403 7464 raft_consensus.cc:383] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.571740 7464 raft_consensus.cc:738] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.572607 7464 consensus_queue.cc:260] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.574486 7453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.46.148:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:33633
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.46.189:36417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.46.189
--webserver_port=37313
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6785
TSAN enabled
I20250626 19:58:27.576395 7453 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 19:58:27.578732 7453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 19:58:27.582331 7464 ts_tablet_manager.cc:1428] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250626 19:58:27.585913 7464 tablet_bootstrap.cc:492] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
W20250626 19:58:27.596499 6711 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
I20250626 19:58:27.600574 7498 raft_consensus.cc:513] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.602803 7464 tablet_bootstrap.cc:654] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:27.609908 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet 734d9d3331b44fc49e1f64b8caf1e4d8 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:27.612177 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet 74b320a2d10a490f90c0ef49c802e4fa from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:27.613037 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 09c15f0a651046a8922074134769806e with delete_type TABLET_DATA_DELETED (replaced by tablet d17c208f1d4b46649145e1d0de155c5d) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:27.613920 7472 tablet_replica.cc:331] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:27.614722 7472 raft_consensus.cc:2241] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:27.615213 7472 raft_consensus.cc:2270] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:27.617573 7472 ts_tablet_manager.cc:1905] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:27.619693 7464 tablet_bootstrap.cc:492] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:27.620153 7464 ts_tablet_manager.cc:1397] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.006s
I20250626 19:58:27.622336 7464 raft_consensus.cc:357] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.628110 7464 raft_consensus.cc:383] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.628437 7464 raft_consensus.cc:738] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.630108 7464 consensus_queue.cc:260] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.636025 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 74b320a2d10a490f90c0ef49c802e4fa
I20250626 19:58:27.636715 7464 ts_tablet_manager.cc:1428] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.016s user 0.003s sys 0.002s
I20250626 19:58:27.637748 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b320a2d10a490f90c0ef49c802e4fa with delete_type TABLET_DATA_DELETED (replaced by tablet 0ce829f8ef704a069f37663c81b64c53) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:27.638414 6711 catalog_manager.cc:6652] ReplaceTablet: tablet 74b320a2d10a490f90c0ef49c802e4fa deleted and replaced by tablet 0ce829f8ef704a069f37663c81b64c53
I20250626 19:58:27.638926 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b320a2d10a490f90c0ef49c802e4fa with delete_type TABLET_DATA_DELETED (replaced by tablet 0ce829f8ef704a069f37663c81b64c53) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:27.639626 7471 tablet_replica.cc:331] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:27.639940 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b320a2d10a490f90c0ef49c802e4fa with delete_type TABLET_DATA_DELETED (replaced by tablet 0ce829f8ef704a069f37663c81b64c53) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:27.640367 7471 raft_consensus.cc:2241] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:27.640937 7471 raft_consensus.cc:2270] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:27.645063 7471 ts_tablet_manager.cc:1905] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:27.645423 7466 tablet_bootstrap.cc:654] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:27.658138 7473 ts_tablet_manager.cc:1918] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:27.660601 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec11cee9ebe94835923f8ecb1e856bfe" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:27.661140 7216 raft_consensus.cc:3058] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:27.667744 7471 ts_tablet_manager.cc:1918] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:27.668246 7471 log.cc:1199] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/74b320a2d10a490f90c0ef49c802e4fa
I20250626 19:58:27.668991 7471 ts_tablet_manager.cc:1939] T 74b320a2d10a490f90c0ef49c802e4fa P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:27.671578 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 74b320a2d10a490f90c0ef49c802e4fa (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:27.676276 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 0ce829f8ef704a069f37663c81b64c53 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:27.677387 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ce829f8ef704a069f37663c81b64c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:27.681061 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 0ce829f8ef704a069f37663c81b64c53 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:27.682123 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ce829f8ef704a069f37663c81b64c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:27.688405 7498 leader_election.cc:290] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Requested vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:27.681336 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 0ce829f8ef704a069f37663c81b64c53 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:27.689872 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec11cee9ebe94835923f8ecb1e856bfe" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:27.690514 7473 log.cc:1199] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/09c15f0a651046a8922074134769806e
I20250626 19:58:27.690605 7361 raft_consensus.cc:3058] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:27.691257 7473 ts_tablet_manager.cc:1939] T 09c15f0a651046a8922074134769806e P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:27.694037 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 09c15f0a651046a8922074134769806e (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
W20250626 19:58:27.695035 6705 catalog_manager.cc:4908] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 74b320a2d10a490f90c0ef49c802e4fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74b320a2d10a490f90c0ef49c802e4fa already in progress: creating tablet
I20250626 19:58:27.690215 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ce829f8ef704a069f37663c81b64c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:27.701781 7467 tablet_bootstrap.cc:492] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:27.703738 7216 raft_consensus.cc:2466] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:27.704944 7006 leader_election.cc:304] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:27.705755 7498 raft_consensus.cc:2802] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:27.706521 7498 raft_consensus.cc:695] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Becoming Leader. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Running, Role: LEADER
I20250626 19:58:27.714828 7467 ts_tablet_manager.cc:1397] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.282s user 0.012s sys 0.000s
I20250626 19:58:27.707387 7498 consensus_queue.cc:237] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.717584 7467 raft_consensus.cc:357] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.718326 7467 raft_consensus.cc:383] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.720666 7472 ts_tablet_manager.cc:1918] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:27.724243 7467 raft_consensus.cc:738] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.715117 7361 raft_consensus.cc:2466] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:27.731388 7467 consensus_queue.cc:260] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.738368 7467 ts_tablet_manager.cc:1428] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20250626 19:58:27.744556 7464 tablet_bootstrap.cc:492] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:27.752295 7464 tablet_bootstrap.cc:654] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:27.758231 7472 log.cc:1199] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/09c15f0a651046a8922074134769806e
I20250626 19:58:27.759183 7472 ts_tablet_manager.cc:1939] T 09c15f0a651046a8922074134769806e P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:27.760398 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b320a2d10a490f90c0ef49c802e4fa with delete_type TABLET_DATA_DELETED (replaced by tablet 0ce829f8ef704a069f37663c81b64c53) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:27.761127 7473 tablet_replica.cc:331] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:27.761858 7473 raft_consensus.cc:2241] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:27.762341 7473 raft_consensus.cc:2270] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:27.763463 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 09c15f0a651046a8922074134769806e (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:27.763752 7466 tablet_bootstrap.cc:492] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:27.764142 7466 ts_tablet_manager.cc:1397] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.221s user 0.012s sys 0.000s
W20250626 19:58:27.764122 6703 catalog_manager.cc:4908] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 74b320a2d10a490f90c0ef49c802e4fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74b320a2d10a490f90c0ef49c802e4fa already in progress: creating tablet
I20250626 19:58:27.770095 7473 ts_tablet_manager.cc:1905] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:58:27.777086 7513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:58:27.779750 7466 raft_consensus.cc:357] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.780421 7466 raft_consensus.cc:383] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.780673 7466 raft_consensus.cc:738] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.771582 7467 tablet_bootstrap.cc:492] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:27.781282 7466 consensus_queue.cc:260] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.784256 6711 catalog_manager.cc:5582] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca reported cstate change: term changed from 0 to 1, leader changed from <none> to bbe151be01e24c82854d9c8c79c797ca (127.4.46.129). New cstate: current_term: 1 leader_uuid: "bbe151be01e24c82854d9c8c79c797ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:27.795639 7464 tablet_bootstrap.cc:492] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:27.796144 7464 ts_tablet_manager.cc:1397] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.004s
I20250626 19:58:27.798382 7464 raft_consensus.cc:357] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.799752 7464 raft_consensus.cc:383] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.800233 7464 raft_consensus.cc:738] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.800954 7464 consensus_queue.cc:260] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.801363 7466 ts_tablet_manager.cc:1428] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.037s user 0.006s sys 0.000s
I20250626 19:58:27.802301 7466 tablet_bootstrap.cc:492] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:27.816413 7464 ts_tablet_manager.cc:1428] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20250626 19:58:27.822997 7464 tablet_bootstrap.cc:492] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:27.829144 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b320a2d10a490f90c0ef49c802e4fa with delete_type TABLET_DATA_DELETED (replaced by tablet 0ce829f8ef704a069f37663c81b64c53) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:27.829313 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 7dcdf6675ca24b7e93c7d545f066efc2 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:27.830076 7472 tablet_replica.cc:331] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:27.831090 7472 raft_consensus.cc:2241] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:27.831580 7472 raft_consensus.cc:2270] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:27.833920 7472 ts_tablet_manager.cc:1905] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:27.850045 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet ec11cee9ebe94835923f8ecb1e856bfe from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:27.855883 7467 tablet_bootstrap.cc:654] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:27.855998 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 7dcdf6675ca24b7e93c7d545f066efc2
I20250626 19:58:27.857771 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 7dcdf6675ca24b7e93c7d545f066efc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 17c8226e69264250b90b1a36a6178aba) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:27.859805 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 7dcdf6675ca24b7e93c7d545f066efc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 17c8226e69264250b90b1a36a6178aba) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:27.860515 7471 tablet_replica.cc:331] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:27.861207 7471 raft_consensus.cc:2241] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:27.861683 7471 raft_consensus.cc:2270] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:27.863976 7471 ts_tablet_manager.cc:1905] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:27.867424 7466 tablet_bootstrap.cc:654] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:27.873086 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 7dcdf6675ca24b7e93c7d545f066efc2 deleted and replaced by tablet 17c8226e69264250b90b1a36a6178aba
I20250626 19:58:27.873229 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 7dcdf6675ca24b7e93c7d545f066efc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 17c8226e69264250b90b1a36a6178aba) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:27.887408 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet ec11cee9ebe94835923f8ecb1e856bfe
I20250626 19:58:27.889321 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet ec11cee9ebe94835923f8ecb1e856bfe with delete_type TABLET_DATA_DELETED (replaced by tablet 1d7a74392e9d475588838a811011e9d6) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:27.890625 6711 catalog_manager.cc:6652] ReplaceTablet: tablet ec11cee9ebe94835923f8ecb1e856bfe deleted and replaced by tablet 1d7a74392e9d475588838a811011e9d6
I20250626 19:58:27.898380 7473 ts_tablet_manager.cc:1918] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:27.898860 7473 log.cc:1199] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/74b320a2d10a490f90c0ef49c802e4fa
I20250626 19:58:27.899819 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet ec11cee9ebe94835923f8ecb1e856bfe with delete_type TABLET_DATA_DELETED (replaced by tablet 1d7a74392e9d475588838a811011e9d6) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:27.900591 7473 ts_tablet_manager.cc:1939] T 74b320a2d10a490f90c0ef49c802e4fa P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:27.911978 7464 tablet_bootstrap.cc:654] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:27.901043 7467 tablet_bootstrap.cc:492] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:27.915180 7467 ts_tablet_manager.cc:1397] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.144s user 0.011s sys 0.000s
I20250626 19:58:27.917368 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 74b320a2d10a490f90c0ef49c802e4fa (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:27.917354 7467 raft_consensus.cc:357] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.918077 7467 raft_consensus.cc:383] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.918325 7467 raft_consensus.cc:738] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
W20250626 19:58:27.918408 6705 catalog_manager.cc:4908] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 7dcdf6675ca24b7e93c7d545f066efc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dcdf6675ca24b7e93c7d545f066efc2 already in progress: creating tablet
I20250626 19:58:27.903054 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet ec11cee9ebe94835923f8ecb1e856bfe with delete_type TABLET_DATA_DELETED (replaced by tablet 1d7a74392e9d475588838a811011e9d6) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:27.919373 7467 consensus_queue.cc:260] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.921597 7467 ts_tablet_manager.cc:1428] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250626 19:58:27.922483 7467 tablet_bootstrap.cc:492] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:27.923820 7472 ts_tablet_manager.cc:1918] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:27.927577 7473 tablet_replica.cc:331] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:27.927732 7466 tablet_bootstrap.cc:492] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:27.931063 7466 ts_tablet_manager.cc:1397] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.129s user 0.010s sys 0.001s
I20250626 19:58:27.931471 7473 raft_consensus.cc:2241] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:27.933408 7466 raft_consensus.cc:357] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.934092 7466 raft_consensus.cc:383] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.939250 7473 raft_consensus.cc:2270] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:27.941150 7466 raft_consensus.cc:738] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.934480 7472 log.cc:1199] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/74b320a2d10a490f90c0ef49c802e4fa
I20250626 19:58:27.943512 7472 ts_tablet_manager.cc:1939] T 74b320a2d10a490f90c0ef49c802e4fa P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:27.932209 7467 tablet_bootstrap.cc:654] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:27.944811 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 7dcdf6675ca24b7e93c7d545f066efc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 17c8226e69264250b90b1a36a6178aba) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:27.948498 7473 ts_tablet_manager.cc:1905] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:27.947268 7466 consensus_queue.cc:260] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.950512 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 74b320a2d10a490f90c0ef49c802e4fa (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:27.953276 7471 ts_tablet_manager.cc:1918] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250626 19:58:27.958489 6703 catalog_manager.cc:4908] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 7dcdf6675ca24b7e93c7d545f066efc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dcdf6675ca24b7e93c7d545f066efc2 already in progress: creating tablet
I20250626 19:58:27.957235 7466 ts_tablet_manager.cc:1428] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20250626 19:58:27.959995 7472 tablet_replica.cc:331] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:27.961107 7471 log.cc:1199] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/7dcdf6675ca24b7e93c7d545f066efc2
I20250626 19:58:27.961887 7471 ts_tablet_manager.cc:1939] T 7dcdf6675ca24b7e93c7d545f066efc2 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:27.960780 7464 tablet_bootstrap.cc:492] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:27.964438 7464 ts_tablet_manager.cc:1397] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.142s user 0.009s sys 0.000s
I20250626 19:58:27.963536 7466 tablet_bootstrap.cc:492] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:27.970156 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 7dcdf6675ca24b7e93c7d545f066efc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 17c8226e69264250b90b1a36a6178aba) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:27.971845 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 17c8226e69264250b90b1a36a6178aba (DEFAULT_TABLE table=default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:27.972906 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17c8226e69264250b90b1a36a6178aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:27.973395 7464 raft_consensus.cc:357] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:27.974028 7464 raft_consensus.cc:383] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:27.974264 7464 raft_consensus.cc:738] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:27.976903 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 17c8226e69264250b90b1a36a6178aba (DEFAULT_TABLE table=default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:27.977986 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17c8226e69264250b90b1a36a6178aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:27.989144 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 17c8226e69264250b90b1a36a6178aba (DEFAULT_TABLE table=default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:27.990348 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17c8226e69264250b90b1a36a6178aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:27.997324 7467 tablet_bootstrap.cc:492] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:27.997776 7467 ts_tablet_manager.cc:1397] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.075s user 0.018s sys 0.000s
I20250626 19:58:27.966542 7472 raft_consensus.cc:2241] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:28.002008 7472 pending_rounds.cc:70] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca: Trying to abort 1 pending ops.
I20250626 19:58:28.002378 7472 pending_rounds.cc:77] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964550011990016 op_type: NO_OP noop_request { }
I20250626 19:58:28.004905 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 7dcdf6675ca24b7e93c7d545f066efc2 (table default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:28.005821 7464 consensus_queue.cc:260] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.002954 7472 raft_consensus.cc:2887] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:28.015116 7472 raft_consensus.cc:2270] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:28.017866 7472 ts_tablet_manager.cc:1905] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:28.020931 7464 ts_tablet_manager.cc:1428] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.050s user 0.004s sys 0.001s
I20250626 19:58:28.030093 7466 tablet_bootstrap.cc:654] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:28.011461 7471 tablet_replica.cc:331] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:28.032318 7471 raft_consensus.cc:2241] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:28.032963 7471 raft_consensus.cc:2270] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:27.999986 7467 raft_consensus.cc:357] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.033928 7467 raft_consensus.cc:383] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:28.034196 7467 raft_consensus.cc:738] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:28.034783 7467 consensus_queue.cc:260] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.036682 7467 ts_tablet_manager.cc:1428] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.039s user 0.004s sys 0.000s
I20250626 19:58:28.037482 7467 tablet_bootstrap.cc:492] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:28.051950 7471 ts_tablet_manager.cc:1905] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:28.066418 7464 tablet_bootstrap.cc:492] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:28.067257 7466 tablet_bootstrap.cc:492] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:28.067639 7466 ts_tablet_manager.cc:1397] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.104s user 0.008s sys 0.001s
I20250626 19:58:28.067567 7473 ts_tablet_manager.cc:1918] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:28.067955 7473 log.cc:1199] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/ec11cee9ebe94835923f8ecb1e856bfe
I20250626 19:58:28.068612 7473 ts_tablet_manager.cc:1939] T ec11cee9ebe94835923f8ecb1e856bfe P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:28.069818 7466 raft_consensus.cc:357] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.070380 7466 raft_consensus.cc:383] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:28.070603 7466 raft_consensus.cc:738] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:28.070816 7473 tablet_replica.cc:331] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:28.071102 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet ec11cee9ebe94835923f8ecb1e856bfe (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
I20250626 19:58:28.071146 7466 consensus_queue.cc:260] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.072926 7464 tablet_bootstrap.cc:654] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:28.079535 7466 ts_tablet_manager.cc:1428] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250626 19:58:28.083353 7466 tablet_bootstrap.cc:492] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:28.094265 7467 tablet_bootstrap.cc:654] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:28.100838 7466 tablet_bootstrap.cc:654] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:28.111053 7498 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:28.111502 7498 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.113397 7498 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:28.114161 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet 0ce829f8ef704a069f37663c81b64c53 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:28.115566 7472 ts_tablet_manager.cc:1918] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:28.126892 7464 tablet_bootstrap.cc:492] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:28.127391 7464 ts_tablet_manager.cc:1397] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.001s
I20250626 19:58:28.129709 7464 raft_consensus.cc:357] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:28.115911 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 19:58:28.121806 7498 raft_consensus.cc:491] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:28.129189 7466 tablet_bootstrap.cc:492] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:28.130735 7464 raft_consensus.cc:383] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:28.131131 7464 raft_consensus.cc:738] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:28.131204 7466 ts_tablet_manager.cc:1397] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.003s
I20250626 19:58:28.131104 7498 raft_consensus.cc:513] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.132335 7464 consensus_queue.cc:260] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.133056 7498 leader_election.cc:290] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:28.133447 7466 raft_consensus.cc:357] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:28.135949 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 19:58:28.140154 7466 raft_consensus.cc:383] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:28.140445 7466 raft_consensus.cc:738] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:28.141731 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
W20250626 19:58:28.137164 7006 leader_election.cc:343] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: 7dcdf6675ca24b7e93c7d545f066efc2
I20250626 19:58:28.145140 7498 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:28.141042 7466 consensus_queue.cc:260] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.146677 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7dcdf6675ca24b7e93c7d545f066efc2" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:28.147346 7361 raft_consensus.cc:2466] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:28.148152 7464 ts_tablet_manager.cc:1428] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.020s user 0.000s sys 0.004s
I20250626 19:58:28.149371 7466 ts_tablet_manager.cc:1428] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.018s user 0.004s sys 0.001s
I20250626 19:58:28.150125 7466 tablet_bootstrap.cc:492] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:28.152176 7471 ts_tablet_manager.cc:1918] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:28.152580 7471 log.cc:1199] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/ec11cee9ebe94835923f8ecb1e856bfe
I20250626 19:58:28.153235 7471 ts_tablet_manager.cc:1939] T ec11cee9ebe94835923f8ecb1e856bfe P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:28.154407 7009 leader_election.cc:304] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters: 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:28.158560 7498 raft_consensus.cc:2802] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:28.158924 7498 raft_consensus.cc:491] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:28.159240 7498 raft_consensus.cc:3058] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:28.163365 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 0ce829f8ef704a069f37663c81b64c53
I20250626 19:58:28.165268 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ce829f8ef704a069f37663c81b64c53 with delete_type TABLET_DATA_DELETED (replaced by tablet 8cce294daa0642aebaf570ad84cfe3c9) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:28.165612 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ce829f8ef704a069f37663c81b64c53 with delete_type TABLET_DATA_DELETED (replaced by tablet 8cce294daa0642aebaf570ad84cfe3c9) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:28.166290 6711 catalog_manager.cc:6652] ReplaceTablet: tablet 0ce829f8ef704a069f37663c81b64c53 deleted and replaced by tablet 8cce294daa0642aebaf570ad84cfe3c9
I20250626 19:58:28.166764 7498 raft_consensus.cc:513] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.167750 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ce829f8ef704a069f37663c81b64c53 with delete_type TABLET_DATA_DELETED (replaced by tablet 8cce294daa0642aebaf570ad84cfe3c9) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:28.168386 7471 tablet_replica.cc:331] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:28.168480 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet ec11cee9ebe94835923f8ecb1e856bfe (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
I20250626 19:58:28.168561 7498 leader_election.cc:290] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Requested vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:28.169092 7471 raft_consensus.cc:2241] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:28.169507 7471 raft_consensus.cc:2270] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250626 19:58:28.170248 7006 leader_election.cc:343] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: 7dcdf6675ca24b7e93c7d545f066efc2
I20250626 19:58:28.171701 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7dcdf6675ca24b7e93c7d545f066efc2" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:28.171587 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 1d7a74392e9d475588838a811011e9d6 (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:28.172241 7361 raft_consensus.cc:3058] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:28.172926 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7a74392e9d475588838a811011e9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:28.177696 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 1d7a74392e9d475588838a811011e9d6 (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:28.178997 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7a74392e9d475588838a811011e9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:28.179136 7466 tablet_bootstrap.cc:654] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:28.185703 7471 ts_tablet_manager.cc:1905] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:28.137773 7472 log.cc:1199] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/ec11cee9ebe94835923f8ecb1e856bfe
I20250626 19:58:28.187398 7472 ts_tablet_manager.cc:1939] T ec11cee9ebe94835923f8ecb1e856bfe P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:28.190528 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 1d7a74392e9d475588838a811011e9d6 (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:28.191864 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7a74392e9d475588838a811011e9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:28.192448 7361 raft_consensus.cc:2466] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:28.193816 7472 tablet_replica.cc:331] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:28.195406 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet ec11cee9ebe94835923f8ecb1e856bfe (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
I20250626 19:58:28.196732 7009 leader_election.cc:304] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters: 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:28.197629 7498 raft_consensus.cc:2802] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:28.198071 7498 raft_consensus.cc:695] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Becoming Leader. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Running, Role: LEADER
I20250626 19:58:28.199164 7498 consensus_queue.cc:237] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.213418 7466 tablet_bootstrap.cc:492] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:28.213830 7466 ts_tablet_manager.cc:1397] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.005s
I20250626 19:58:28.216619 7475 raft_consensus.cc:491] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:28.217109 7475 raft_consensus.cc:513] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.220378 7473 raft_consensus.cc:2241] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:28.226972 7473 raft_consensus.cc:2270] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:28.222611 7475 leader_election.cc:290] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:28.231156 7472 raft_consensus.cc:2241] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:28.233635 7472 pending_rounds.cc:70] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca: Trying to abort 1 pending ops.
I20250626 19:58:28.233930 7472 pending_rounds.cc:77] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964551992614912 op_type: NO_OP noop_request { }
I20250626 19:58:28.234364 7472 raft_consensus.cc:2887] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:28.234261 7466 raft_consensus.cc:357] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.234930 7466 raft_consensus.cc:383] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:28.235201 7466 raft_consensus.cc:738] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:28.235747 7466 consensus_queue.cc:260] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.236977 7472 raft_consensus.cc:2270] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:28.252125 7466 ts_tablet_manager.cc:1428] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.038s user 0.005s sys 0.001s
I20250626 19:58:28.252959 7472 ts_tablet_manager.cc:1905] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:28.263463 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "734d9d3331b44fc49e1f64b8caf1e4d8" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:28.264423 7361 raft_consensus.cc:2391] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 315d304854c541bf8b5e6b1eea712d4a in current term 1: Already voted for candidate bbe151be01e24c82854d9c8c79c797ca in this term.
I20250626 19:58:28.268566 7473 ts_tablet_manager.cc:1905] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:28.281785 7472 ts_tablet_manager.cc:1918] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:28.286475 7472 log.cc:1199] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/7dcdf6675ca24b7e93c7d545f066efc2
I20250626 19:58:28.287370 7472 ts_tablet_manager.cc:1939] T 7dcdf6675ca24b7e93c7d545f066efc2 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:28.295284 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 7dcdf6675ca24b7e93c7d545f066efc2 (table default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:28.296525 7467 tablet_bootstrap.cc:492] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:28.296298 7472 tablet_replica.cc:331] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:28.296950 7467 ts_tablet_manager.cc:1397] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.260s user 0.018s sys 0.012s
I20250626 19:58:28.297571 7472 raft_consensus.cc:2241] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:28.298123 7472 raft_consensus.cc:2270] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:28.299258 7467 raft_consensus.cc:357] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.299883 7467 raft_consensus.cc:383] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:28.300146 7467 raft_consensus.cc:738] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:28.300642 7472 ts_tablet_manager.cc:1905] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:28.300892 7467 consensus_queue.cc:260] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.308894 7467 ts_tablet_manager.cc:1428] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20250626 19:58:28.315320 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "734d9d3331b44fc49e1f64b8caf1e4d8" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:28.317128 7150 leader_election.cc:304] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a; no voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:28.318223 7475 raft_consensus.cc:3058] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:28.322326 7471 ts_tablet_manager.cc:1918] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:28.338591 7472 ts_tablet_manager.cc:1918] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:28.339152 7472 log.cc:1199] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/0ce829f8ef704a069f37663c81b64c53
I20250626 19:58:28.339887 7472 ts_tablet_manager.cc:1939] T 0ce829f8ef704a069f37663c81b64c53 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:28.342078 7467 tablet_bootstrap.cc:492] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:28.342512 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 0ce829f8ef704a069f37663c81b64c53 (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:28.343109 7475 raft_consensus.cc:2747] T 734d9d3331b44fc49e1f64b8caf1e4d8 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:28.322687 7471 log.cc:1199] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/0ce829f8ef704a069f37663c81b64c53
I20250626 19:58:28.359690 7471 ts_tablet_manager.cc:1939] T 0ce829f8ef704a069f37663c81b64c53 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:28.361929 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 0ce829f8ef704a069f37663c81b64c53 (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:28.366255 7467 tablet_bootstrap.cc:654] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:28.387354 6716 catalog_manager.cc:3460] renamed table default.table_35326ce5948a4f5ab528b8243ecb7479 [id=eff2225412944029aed88f6c87744fc8] in HMS: new name default.table_a5221df6a6b54bcdafa382fca51b613d
I20250626 19:58:28.397374 7467 tablet_bootstrap.cc:492] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:28.398144 7467 ts_tablet_manager.cc:1397] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.000s
I20250626 19:58:28.401750 7467 raft_consensus.cc:357] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.409200 7467 raft_consensus.cc:383] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:28.409631 7467 raft_consensus.cc:738] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:28.410665 7467 consensus_queue.cc:260] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.417622 7475 raft_consensus.cc:491] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:28.418074 7475 raft_consensus.cc:513] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.421679 7467 ts_tablet_manager.cc:1428] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20250626 19:58:28.422546 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d17c208f1d4b46649145e1d0de155c5d" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:28.423126 7361 raft_consensus.cc:2466] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:28.424355 7152 leader_election.cc:304] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:28.427662 7475 leader_election.cc:290] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:28.428139 7475 raft_consensus.cc:2802] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:28.428452 7475 raft_consensus.cc:491] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:28.428766 7475 raft_consensus.cc:3058] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:28.456503 6944 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17c8226e69264250b90b1a36a6178aba consensus state has no leader
W20250626 19:58:28.460707 6714 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f9f594403d34451f8f2fa6d607d33a2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:28.462740 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ce829f8ef704a069f37663c81b64c53 with delete_type TABLET_DATA_DELETED (replaced by tablet 8cce294daa0642aebaf570ad84cfe3c9) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:28.464352 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 7dcdf6675ca24b7e93c7d545f066efc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 17c8226e69264250b90b1a36a6178aba) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:28.465657 7467 tablet_bootstrap.cc:492] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
W20250626 19:58:28.465467 6713 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_64bb792c0fa74fab8347bf7bda0dedbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:58:28.469779 6715 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5283819977d9487a9d1aaae326249a46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:28.469256 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d17c208f1d4b46649145e1d0de155c5d" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:28.470471 7073 raft_consensus.cc:2466] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:28.473675 7467 tablet_bootstrap.cc:654] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:28.493148 6703 catalog_manager.cc:4895] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 0ce829f8ef704a069f37663c81b64c53 because the tablet was not found. No further retry: Not found: Tablet not found: 0ce829f8ef704a069f37663c81b64c53
W20250626 19:58:28.494377 6703 catalog_manager.cc:4895] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 7dcdf6675ca24b7e93c7d545f066efc2 because the tablet was not found. No further retry: Not found: Tablet not found: 7dcdf6675ca24b7e93c7d545f066efc2
I20250626 19:58:28.497749 7464 tablet_bootstrap.cc:492] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:28.506875 7464 tablet_bootstrap.cc:654] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:28.510797 7152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.131:33703, user_credentials={real_user=slave}} blocked reactor thread for 86607.9us
I20250626 19:58:28.516263 7475 raft_consensus.cc:513] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.517650 7467 tablet_bootstrap.cc:492] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:28.518119 7467 ts_tablet_manager.cc:1397] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.000s
I20250626 19:58:28.519343 7466 tablet_bootstrap.cc:492] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:28.520582 7467 raft_consensus.cc:357] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.521231 7467 raft_consensus.cc:383] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:28.521483 7467 raft_consensus.cc:738] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:28.522096 7467 consensus_queue.cc:260] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:28.522490 6704 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 17c8226e69264250b90b1a36a6178aba (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 17c8226e69264250b90b1a36a6178aba consensus state has no leader
I20250626 19:58:28.526212 6711 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 17c8226e69264250b90b1a36a6178aba (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:28.527525 7466 tablet_bootstrap.cc:654] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:28.536756 7467 ts_tablet_manager.cc:1428] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.018s user 0.005s sys 0.002s
W20250626 19:58:28.536777 6704 consensus_peers.cc:487] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f -> Peer 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417): Couldn't send request to peer 8ae38ec946e14198ae4adc8dde9d66e8. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250626 19:58:28.539822 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet ae04997b77924e8ba479c5c34ac822d7 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:28.541989 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 734d9d3331b44fc49e1f64b8caf1e4d8 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:28.542703 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d17c208f1d4b46649145e1d0de155c5d" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:28.543246 7073 raft_consensus.cc:3058] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:28.548620 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 8cce294daa0642aebaf570ad84cfe3c9 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:28.551743 7073 raft_consensus.cc:2466] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:28.552006 7464 tablet_bootstrap.cc:492] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:28.552874 7150 leader_election.cc:304] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:28.553977 7536 raft_consensus.cc:2802] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:28.555711 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cce294daa0642aebaf570ad84cfe3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:28.555627 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 8cce294daa0642aebaf570ad84cfe3c9 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:28.556871 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cce294daa0642aebaf570ad84cfe3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:28.560111 7464 ts_tablet_manager.cc:1397] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.004s
I20250626 19:58:28.532469 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 8cce294daa0642aebaf570ad84cfe3c9 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:28.561070 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cce294daa0642aebaf570ad84cfe3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:28.562415 7464 raft_consensus.cc:357] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.564929 7464 raft_consensus.cc:383] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:28.565176 7464 raft_consensus.cc:738] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:28.567051 7466 tablet_bootstrap.cc:492] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:28.567916 7466 ts_tablet_manager.cc:1397] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.003s
I20250626 19:58:28.572477 7536 raft_consensus.cc:695] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:28.573827 7536 consensus_queue.cc:237] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.577600 7464 consensus_queue.cc:260] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.581538 7466 raft_consensus.cc:357] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.582856 7466 raft_consensus.cc:383] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:28.592049 6709 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 17c8226e69264250b90b1a36a6178aba (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:28.592829 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d17c208f1d4b46649145e1d0de155c5d" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:28.593621 7361 raft_consensus.cc:3058] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:28.612181 7475 leader_election.cc:290] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:28.619547 7361 raft_consensus.cc:2466] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:28.636363 7464 ts_tablet_manager.cc:1428] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.076s user 0.014s sys 0.007s
I20250626 19:58:28.638291 7473 ts_tablet_manager.cc:1918] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:28.638864 7473 log.cc:1199] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/7dcdf6675ca24b7e93c7d545f066efc2
I20250626 19:58:28.639683 7473 ts_tablet_manager.cc:1939] T 7dcdf6675ca24b7e93c7d545f066efc2 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:28.588419 7498 consensus_queue.cc:1035] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:28.642726 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 7dcdf6675ca24b7e93c7d545f066efc2 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:28.647166 7543 raft_consensus.cc:491] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:28.647730 7543 raft_consensus.cc:513] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.648561 7477 consensus_queue.cc:1035] T 734d9d3331b44fc49e1f64b8caf1e4d8 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:28.650137 7466 raft_consensus.cc:738] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:28.650684 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17c8226e69264250b90b1a36a6178aba" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:28.651291 7073 raft_consensus.cc:2466] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:28.650974 7466 consensus_queue.cc:260] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.652439 7150 leader_election.cc:304] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:28.653488 7466 ts_tablet_manager.cc:1428] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.083s user 0.006s sys 0.000s
I20250626 19:58:28.657287 7475 raft_consensus.cc:2802] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:28.657694 7475 raft_consensus.cc:491] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:28.658044 7475 raft_consensus.cc:3058] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:28.668643 6711 catalog_manager.cc:5582] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130). New cstate: current_term: 1 leader_uuid: "315d304854c541bf8b5e6b1eea712d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:28.698921 7464 tablet_bootstrap.cc:492] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:28.705591 7464 tablet_bootstrap.cc:654] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:28.712391 7466 tablet_bootstrap.cc:492] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:28.719508 7466 tablet_bootstrap.cc:654] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:28.725826 7499 raft_consensus.cc:491] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:28.726536 7499 raft_consensus.cc:513] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.729576 7499 leader_election.cc:290] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:28.730822 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d7a74392e9d475588838a811011e9d6" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:28.731724 7215 raft_consensus.cc:2466] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:28.733075 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17c8226e69264250b90b1a36a6178aba" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:28.733820 7361 raft_consensus.cc:2466] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:28.735103 7293 leader_election.cc:304] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:28.736253 7466 tablet_bootstrap.cc:492] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:28.736640 7466 ts_tablet_manager.cc:1397] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.000s
I20250626 19:58:28.737367 7499 raft_consensus.cc:2802] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:28.737871 7499 raft_consensus.cc:491] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:28.738319 7499 raft_consensus.cc:3058] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:28.740506 7464 tablet_bootstrap.cc:492] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:28.740921 7464 ts_tablet_manager.cc:1397] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.000s
I20250626 19:58:28.741279 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d7a74392e9d475588838a811011e9d6" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:28.742832 7073 raft_consensus.cc:2466] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:28.745198 7464 raft_consensus.cc:357] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.745849 7464 raft_consensus.cc:383] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:28.746114 7464 raft_consensus.cc:738] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:28.748555 7466 raft_consensus.cc:357] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.749272 7466 raft_consensus.cc:383] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:28.749526 7466 raft_consensus.cc:738] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:28.750106 7466 consensus_queue.cc:260] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.751201 7499 raft_consensus.cc:513] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.758859 7543 leader_election.cc:290] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:28.760777 7464 consensus_queue.cc:260] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:28.765071 6709 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
I20250626 19:58:28.776921 7466 ts_tablet_manager.cc:1428] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.040s user 0.003s sys 0.004s
I20250626 19:58:28.771858 7464 ts_tablet_manager.cc:1428] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.031s user 0.006s sys 0.001s
W20250626 19:58:28.797863 6711 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
I20250626 19:58:28.805231 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d7a74392e9d475588838a811011e9d6" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:28.805769 7073 raft_consensus.cc:3058] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:28.812951 7073 raft_consensus.cc:2466] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:28.814205 7294 leader_election.cc:304] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:28.795454 7473 tablet_replica.cc:331] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:28.821416 7473 raft_consensus.cc:2241] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:28.844146 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d7a74392e9d475588838a811011e9d6" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:28.844895 7216 raft_consensus.cc:3058] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:28.852723 7216 raft_consensus.cc:2466] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:28.855777 7473 raft_consensus.cc:2270] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:28.864529 7473 ts_tablet_manager.cc:1905] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:28.865608 7475 raft_consensus.cc:513] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.872282 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17c8226e69264250b90b1a36a6178aba" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:28.873106 7073 raft_consensus.cc:3058] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:28.875931 7543 raft_consensus.cc:491] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:28.876355 7543 raft_consensus.cc:513] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.878060 7543 leader_election.cc:290] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:28.879937 7072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cce294daa0642aebaf570ad84cfe3c9" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:28.880555 7072 raft_consensus.cc:2466] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:28.881795 7150 leader_election.cc:304] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:28.882504 7543 raft_consensus.cc:2802] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:28.882819 7543 raft_consensus.cc:491] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:28.883096 7543 raft_consensus.cc:3058] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:28.884907 6709 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
W20250626 19:58:28.887408 6709 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
I20250626 19:58:28.891474 7543 raft_consensus.cc:513] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.893985 7072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cce294daa0642aebaf570ad84cfe3c9" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:28.894474 7072 raft_consensus.cc:3058] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:28.859154 7499 leader_election.cc:290] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:28.868613 7554 raft_consensus.cc:2802] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
W20250626 19:58:28.869100 7294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:33129, user_credentials={real_user=slave}} blocked reactor thread for 55012.8us
I20250626 19:58:28.877799 7467 tablet_bootstrap.cc:492] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:28.902102 7072 raft_consensus.cc:2466] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:28.903231 7150 leader_election.cc:304] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:28.904060 7536 raft_consensus.cc:2802] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:28.904449 7536 raft_consensus.cc:695] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:28.905169 7536 consensus_queue.cc:237] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.908504 7467 tablet_bootstrap.cc:654] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:28.912230 7554 raft_consensus.cc:695] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:28.912266 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cce294daa0642aebaf570ad84cfe3c9" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
W20250626 19:58:28.913465 7152 leader_election.cc:343] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:28.913254 7554 consensus_queue.cc:237] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.915763 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cce294daa0642aebaf570ad84cfe3c9" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
W20250626 19:58:28.919894 7152 leader_election.cc:343] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:28.920724 7543 leader_election.cc:290] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:28.928375 7467 tablet_bootstrap.cc:492] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:28.928769 7467 ts_tablet_manager.cc:1397] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.000s
I20250626 19:58:28.929033 7475 leader_election.cc:290] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:28.929977 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17c8226e69264250b90b1a36a6178aba" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:28.930492 7360 raft_consensus.cc:3058] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:28.935989 7073 raft_consensus.cc:2466] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:28.938241 7467 raft_consensus.cc:357] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.938903 7467 raft_consensus.cc:383] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:28.939149 7467 raft_consensus.cc:738] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:28.938915 7360 raft_consensus.cc:2466] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:28.940872 7152 leader_election.cc:304] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:28.942373 7475 raft_consensus.cc:2802] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:28.943164 7475 raft_consensus.cc:695] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:28.942993 7467 consensus_queue.cc:260] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.944249 7475 consensus_queue.cc:237] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:28.931303 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet d17c208f1d4b46649145e1d0de155c5d from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:28.948899 7467 ts_tablet_manager.cc:1428] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
W20250626 19:58:28.995311 6709 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
I20250626 19:58:29.025355 6709 catalog_manager.cc:5582] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130). New cstate: current_term: 1 leader_uuid: "315d304854c541bf8b5e6b1eea712d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:29.032373 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet d17c208f1d4b46649145e1d0de155c5d
I20250626 19:58:29.034849 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet d17c208f1d4b46649145e1d0de155c5d with delete_type TABLET_DATA_DELETED (replaced by tablet 851ca41b87fa4f2e9794a407a3534868) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:29.036425 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet d17c208f1d4b46649145e1d0de155c5d with delete_type TABLET_DATA_DELETED (replaced by tablet 851ca41b87fa4f2e9794a407a3534868) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:29.048556 6711 catalog_manager.cc:6652] ReplaceTablet: tablet d17c208f1d4b46649145e1d0de155c5d deleted and replaced by tablet 851ca41b87fa4f2e9794a407a3534868
I20250626 19:58:29.053184 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet d17c208f1d4b46649145e1d0de155c5d with delete_type TABLET_DATA_DELETED (replaced by tablet 851ca41b87fa4f2e9794a407a3534868) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:29.067076 6711 catalog_manager.cc:5582] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } }
I20250626 19:58:29.080948 7473 ts_tablet_manager.cc:1918] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:29.081372 7473 log.cc:1199] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/0ce829f8ef704a069f37663c81b64c53
I20250626 19:58:29.082275 7473 ts_tablet_manager.cc:1939] T 0ce829f8ef704a069f37663c81b64c53 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:29.085664 7473 tablet_replica.cc:331] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:29.088140 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 0ce829f8ef704a069f37663c81b64c53 (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:29.086690 7473 raft_consensus.cc:2241] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:29.098618 7473 raft_consensus.cc:2270] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:29.100741 7473 ts_tablet_manager.cc:1905] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:29.133944 7559 tablet_replica.cc:331] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:29.135051 7559 raft_consensus.cc:2241] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:29.136584 7560 tablet_replica.cc:331] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:29.137481 7560 raft_consensus.cc:2241] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:29.145543 7560 pending_rounds.cc:70] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a: Trying to abort 1 pending ops.
I20250626 19:58:29.146250 7559 raft_consensus.cc:2270] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:29.146899 7560 pending_rounds.cc:77] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964553534353408 op_type: NO_OP noop_request { }
I20250626 19:58:29.149307 7559 ts_tablet_manager.cc:1905] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:29.162734 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet d17c208f1d4b46649145e1d0de155c5d from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:29.175606 7473 ts_tablet_manager.cc:1918] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:29.176009 7473 log.cc:1199] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/d17c208f1d4b46649145e1d0de155c5d
I20250626 19:58:29.176668 7473 ts_tablet_manager.cc:1939] T d17c208f1d4b46649145e1d0de155c5d P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:29.186990 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet d17c208f1d4b46649145e1d0de155c5d (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:29.197160 7560 raft_consensus.cc:2887] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:29.197679 7560 raft_consensus.cc:2270] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:29.212932 7560 ts_tablet_manager.cc:1905] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:29.216470 6709 catalog_manager.cc:5582] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130). New cstate: current_term: 1 leader_uuid: "315d304854c541bf8b5e6b1eea712d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:29.218024 6709 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 17c8226e69264250b90b1a36a6178aba (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:29.224584 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 851ca41b87fa4f2e9794a407a3534868 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:29.227377 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 851ca41b87fa4f2e9794a407a3534868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:29.233768 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 851ca41b87fa4f2e9794a407a3534868 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:29.234941 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 851ca41b87fa4f2e9794a407a3534868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:29.235023 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 851ca41b87fa4f2e9794a407a3534868 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:29.236598 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 851ca41b87fa4f2e9794a407a3534868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:29.260057 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet ae04997b77924e8ba479c5c34ac822d7 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:29.261130 7467 tablet_bootstrap.cc:492] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:29.268528 7467 tablet_bootstrap.cc:654] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:29.275604 7560 ts_tablet_manager.cc:1918] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:29.276019 7560 log.cc:1199] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/d17c208f1d4b46649145e1d0de155c5d
I20250626 19:58:29.277730 7467 tablet_bootstrap.cc:492] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:29.278143 7467 ts_tablet_manager.cc:1397] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20250626 19:58:29.280488 7467 raft_consensus.cc:357] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.281126 7467 raft_consensus.cc:383] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:29.281381 7467 raft_consensus.cc:738] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:29.282044 7467 consensus_queue.cc:260] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.287420 7467 ts_tablet_manager.cc:1428] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.009s user 0.001s sys 0.006s
I20250626 19:58:29.296810 7560 ts_tablet_manager.cc:1939] T d17c208f1d4b46649145e1d0de155c5d P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:29.300448 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet d17c208f1d4b46649145e1d0de155c5d (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:29.371917 7568 tablet_bootstrap.cc:492] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:29.380184 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet ae04997b77924e8ba479c5c34ac822d7 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:29.391443 7569 tablet_bootstrap.cc:492] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:29.368010 7559 ts_tablet_manager.cc:1918] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:29.399407 7559 log.cc:1199] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/d17c208f1d4b46649145e1d0de155c5d
I20250626 19:58:29.403666 7559 ts_tablet_manager.cc:1939] T d17c208f1d4b46649145e1d0de155c5d P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:29.407078 6717 catalog_manager.cc:3460] renamed table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb] in HMS: new name default.table_252448e828024438ba317b8277b66523
I20250626 19:58:29.418545 7569 tablet_bootstrap.cc:654] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:29.419469 7568 tablet_bootstrap.cc:654] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:29.420500 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet d17c208f1d4b46649145e1d0de155c5d (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:29.447284 7568 tablet_bootstrap.cc:492] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:29.447791 7568 ts_tablet_manager.cc:1397] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.008s
I20250626 19:58:29.450567 7568 raft_consensus.cc:357] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.459182 7569 tablet_bootstrap.cc:492] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:29.459753 7569 ts_tablet_manager.cc:1397] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.069s user 0.013s sys 0.000s
I20250626 19:58:29.457262 7568 raft_consensus.cc:383] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:29.464352 7568 raft_consensus.cc:738] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:29.471457 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 17c8226e69264250b90b1a36a6178aba from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:29.472098 7568 consensus_queue.cc:260] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.472954 7569 raft_consensus.cc:357] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.475234 7568 ts_tablet_manager.cc:1428] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20250626 19:58:29.475574 7569 raft_consensus.cc:383] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:29.477349 7569 raft_consensus.cc:738] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:29.478228 7569 consensus_queue.cc:260] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.508258 7569 ts_tablet_manager.cc:1428] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.048s user 0.005s sys 0.010s
I20250626 19:58:29.520951 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 17c8226e69264250b90b1a36a6178aba
I20250626 19:58:29.522600 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 17c8226e69264250b90b1a36a6178aba with delete_type TABLET_DATA_DELETED (replaced by tablet 180d9cb0d2874a8e9ad6d45fca6cb577) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:29.523434 7559 tablet_replica.cc:331] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:29.524366 7559 raft_consensus.cc:2241] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:29.524894 7559 raft_consensus.cc:2270] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:29.526713 7559 ts_tablet_manager.cc:1905] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:29.530059 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 17c8226e69264250b90b1a36a6178aba with delete_type TABLET_DATA_DELETED (replaced by tablet 180d9cb0d2874a8e9ad6d45fca6cb577) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:29.531003 7560 tablet_replica.cc:331] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:29.531747 7560 raft_consensus.cc:2241] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:29.532452 7560 pending_rounds.cc:70] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a: Trying to abort 1 pending ops.
I20250626 19:58:29.532692 7560 pending_rounds.cc:77] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964555084632064 op_type: NO_OP noop_request { }
I20250626 19:58:29.533051 7560 raft_consensus.cc:2887] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:29.533367 7560 raft_consensus.cc:2270] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:29.535872 7560 ts_tablet_manager.cc:1905] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:29.540786 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 17c8226e69264250b90b1a36a6178aba with delete_type TABLET_DATA_DELETED (replaced by tablet 180d9cb0d2874a8e9ad6d45fca6cb577) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:29.541415 7473 tablet_replica.cc:331] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:29.542073 7473 raft_consensus.cc:2241] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:29.542471 7473 raft_consensus.cc:2270] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:29.544106 7473 ts_tablet_manager.cc:1905] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:29.553423 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:29.554582 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 180d9cb0d2874a8e9ad6d45fca6cb577. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:29.563365 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 17c8226e69264250b90b1a36a6178aba deleted and replaced by tablet 180d9cb0d2874a8e9ad6d45fca6cb577
I20250626 19:58:29.563638 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:29.564604 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 180d9cb0d2874a8e9ad6d45fca6cb577. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:29.561041 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:29.568404 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 180d9cb0d2874a8e9ad6d45fca6cb577. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:29.590260 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 851ca41b87fa4f2e9794a407a3534868 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:29.608803 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 851ca41b87fa4f2e9794a407a3534868
I20250626 19:58:29.613432 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 851ca41b87fa4f2e9794a407a3534868 with delete_type TABLET_DATA_DELETED (replaced by tablet 052c389ae05f47649465c86991795f7f) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:29.615257 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 851ca41b87fa4f2e9794a407a3534868 with delete_type TABLET_DATA_DELETED (replaced by tablet 052c389ae05f47649465c86991795f7f) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:29.621560 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 851ca41b87fa4f2e9794a407a3534868 with delete_type TABLET_DATA_DELETED (replaced by tablet 052c389ae05f47649465c86991795f7f) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:29.625334 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 851ca41b87fa4f2e9794a407a3534868 with delete_type TABLET_DATA_DELETED (replaced by tablet 052c389ae05f47649465c86991795f7f) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:29.631398 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 851ca41b87fa4f2e9794a407a3534868 deleted and replaced by tablet 052c389ae05f47649465c86991795f7f
W20250626 19:58:29.634985 6702 catalog_manager.cc:5001] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): alter failed for tablet 17c8226e69264250b90b1a36a6178aba (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]),no further retry: Illegal state: Tablet not RUNNING: STOPPED
I20250626 19:58:29.638015 7568 tablet_bootstrap.cc:492] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:29.639472 7569 tablet_bootstrap.cc:492] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:29.645572 7569 tablet_bootstrap.cc:654] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:29.652415 7473 ts_tablet_manager.cc:1918] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:29.652788 7473 log.cc:1199] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/17c8226e69264250b90b1a36a6178aba
I20250626 19:58:29.653398 7473 ts_tablet_manager.cc:1939] T 17c8226e69264250b90b1a36a6178aba P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:29.655360 7568 tablet_bootstrap.cc:654] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:29.661103 7467 tablet_bootstrap.cc:492] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:29.661266 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 052c389ae05f47649465c86991795f7f (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:29.662189 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 052c389ae05f47649465c86991795f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:29.665104 7559 ts_tablet_manager.cc:1918] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:29.666395 7467 tablet_bootstrap.cc:654] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:29.668020 7560 ts_tablet_manager.cc:1918] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:29.668421 7560 log.cc:1199] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/17c8226e69264250b90b1a36a6178aba
I20250626 19:58:29.669188 7560 ts_tablet_manager.cc:1939] T 17c8226e69264250b90b1a36a6178aba P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:29.674294 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 052c389ae05f47649465c86991795f7f (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:29.675441 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 052c389ae05f47649465c86991795f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:29.675316 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 052c389ae05f47649465c86991795f7f (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:29.676249 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 052c389ae05f47649465c86991795f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:29.676710 7568 tablet_bootstrap.cc:492] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:29.677095 7568 ts_tablet_manager.cc:1397] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.002s
I20250626 19:58:29.684608 7569 tablet_bootstrap.cc:492] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:29.685025 7569 ts_tablet_manager.cc:1397] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.001s
I20250626 19:58:29.687772 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 17c8226e69264250b90b1a36a6178aba (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:29.688635 7473 tablet_replica.cc:331] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:29.689268 7473 raft_consensus.cc:2241] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:29.689847 7473 raft_consensus.cc:2270] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:29.690992 7568 raft_consensus.cc:357] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.691671 7568 raft_consensus.cc:383] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:29.691735 7473 ts_tablet_manager.cc:1905] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:29.691955 7568 raft_consensus.cc:738] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:29.694883 7554 raft_consensus.cc:491] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:29.665491 7559 log.cc:1199] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/17c8226e69264250b90b1a36a6178aba
I20250626 19:58:29.695410 7554 raft_consensus.cc:513] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.696982 7467 tablet_bootstrap.cc:492] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:29.697357 7467 ts_tablet_manager.cc:1397] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.005s
I20250626 19:58:29.697456 7554 leader_election.cc:290] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:29.698712 7569 raft_consensus.cc:357] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.700018 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cce294daa0642aebaf570ad84cfe3c9" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:29.701371 7568 consensus_queue.cc:260] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.701617 7467 raft_consensus.cc:357] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.702279 7467 raft_consensus.cc:383] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:29.702566 7467 raft_consensus.cc:738] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:29.696743 7559 ts_tablet_manager.cc:1939] T 17c8226e69264250b90b1a36a6178aba P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:29.705349 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 17c8226e69264250b90b1a36a6178aba (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:29.699576 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cce294daa0642aebaf570ad84cfe3c9" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:29.705960 7467 consensus_queue.cc:260] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.706713 7073 raft_consensus.cc:2391] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 297d2c913667408bae82668a557e06c7 in current term 1: Already voted for candidate 315d304854c541bf8b5e6b1eea712d4a in this term.
I20250626 19:58:29.699441 7569 raft_consensus.cc:383] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:29.707729 7569 raft_consensus.cc:738] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:29.708456 7569 consensus_queue.cc:260] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.710705 7569 ts_tablet_manager.cc:1428] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20250626 19:58:29.711601 7559 tablet_replica.cc:331] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:29.712503 7294 leader_election.cc:304] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 297d2c913667408bae82668a557e06c7; no voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:29.712734 7568 ts_tablet_manager.cc:1428] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20250626 19:58:29.716739 6709 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
W20250626 19:58:29.718187 6709 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 180d9cb0d2874a8e9ad6d45fca6cb577 consensus state has no leader
I20250626 19:58:29.718425 7559 raft_consensus.cc:2241] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:29.718874 7559 raft_consensus.cc:2270] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:29.718947 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 17c8226e69264250b90b1a36a6178aba (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:29.721031 6711 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:29.722913 7554 raft_consensus.cc:3058] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:29.727754 6711 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:29.728278 7560 tablet_replica.cc:331] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:29.729041 7560 raft_consensus.cc:2241] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:29.729521 7560 raft_consensus.cc:2270] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:29.732915 7559 ts_tablet_manager.cc:1905] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:29.734709 7554 raft_consensus.cc:2747] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:29.735787 6709 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:29.740806 7467 ts_tablet_manager.cc:1428] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.043s user 0.005s sys 0.000s
I20250626 19:58:29.745141 7467 tablet_bootstrap.cc:492] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:29.750999 6709 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:29.759330 7554 raft_consensus.cc:491] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:29.759779 7554 raft_consensus.cc:513] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.761682 7467 tablet_bootstrap.cc:654] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:29.764961 7560 ts_tablet_manager.cc:1905] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:29.765911 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "180d9cb0d2874a8e9ad6d45fca6cb577" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:29.766533 7073 raft_consensus.cc:2466] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:29.767102 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "180d9cb0d2874a8e9ad6d45fca6cb577" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:29.767836 7216 raft_consensus.cc:2466] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
W20250626 19:58:29.771080 6704 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 180d9cb0d2874a8e9ad6d45fca6cb577 consensus state has no leader
I20250626 19:58:29.771296 7554 leader_election.cc:290] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:29.767707 7294 leader_election.cc:304] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:29.772967 7554 raft_consensus.cc:2802] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:29.773298 7554 raft_consensus.cc:491] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:29.773610 7554 raft_consensus.cc:3058] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:29.775378 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 180d9cb0d2874a8e9ad6d45fca6cb577 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:29.779517 7568 tablet_bootstrap.cc:492] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:29.782367 7569 tablet_bootstrap.cc:492] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:29.785343 7568 tablet_bootstrap.cc:654] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:29.788647 7569 tablet_bootstrap.cc:654] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:29.801934 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 180d9cb0d2874a8e9ad6d45fca6cb577
I20250626 19:58:29.804076 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b9dfd7e091e4d0ea3a698a12a6be585) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:29.804023 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b9dfd7e091e4d0ea3a698a12a6be585) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:29.805325 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 180d9cb0d2874a8e9ad6d45fca6cb577 deleted and replaced by tablet 2b9dfd7e091e4d0ea3a698a12a6be585
I20250626 19:58:29.807129 7467 tablet_bootstrap.cc:492] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:29.807569 7467 ts_tablet_manager.cc:1397] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.000s
I20250626 19:58:29.809791 7467 raft_consensus.cc:357] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.810423 7467 raft_consensus.cc:383] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:29.810621 7475 consensus_queue.cc:1035] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:29.818799 7569 tablet_bootstrap.cc:492] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:29.819234 7569 ts_tablet_manager.cc:1397] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.001s
I20250626 19:58:29.820324 7568 tablet_bootstrap.cc:492] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:29.820736 7568 ts_tablet_manager.cc:1397] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.001s
I20250626 19:58:29.821698 7569 raft_consensus.cc:357] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.822351 7569 raft_consensus.cc:383] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:29.810732 7467 raft_consensus.cc:738] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:29.823722 7467 consensus_queue.cc:260] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.826300 7467 ts_tablet_manager.cc:1428] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.018s user 0.002s sys 0.002s
I20250626 19:58:29.811102 7577 consensus_queue.cc:1035] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20250626 19:58:29.831348 7473 ts_tablet_manager.cc:1918] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:29.831804 7473 log.cc:1199] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/851ca41b87fa4f2e9794a407a3534868
I20250626 19:58:29.817657 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b9dfd7e091e4d0ea3a698a12a6be585) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:29.832417 7473 ts_tablet_manager.cc:1939] T 851ca41b87fa4f2e9794a407a3534868 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:29.813110 7554 raft_consensus.cc:513] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.840708 7072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "180d9cb0d2874a8e9ad6d45fca6cb577" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:29.841243 7072 raft_consensus.cc:3058] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:29.849548 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "180d9cb0d2874a8e9ad6d45fca6cb577" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:29.849725 7554 leader_election.cc:290] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:29.850121 7215 raft_consensus.cc:3058] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:29.840415 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 851ca41b87fa4f2e9794a407a3534868 (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:29.855472 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet ae04997b77924e8ba479c5c34ac822d7 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:29.855475 7072 raft_consensus.cc:2466] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:29.857316 7294 leader_election.cc:304] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:29.858129 7554 raft_consensus.cc:2802] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:29.858577 7554 raft_consensus.cc:695] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:29.859874 7215 raft_consensus.cc:2466] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:29.859804 7554 consensus_queue.cc:237] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.863655 7559 ts_tablet_manager.cc:1918] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:29.864336 7473 tablet_replica.cc:331] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:29.869479 7559 log.cc:1199] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/851ca41b87fa4f2e9794a407a3534868
I20250626 19:58:29.870239 7559 ts_tablet_manager.cc:1939] T 851ca41b87fa4f2e9794a407a3534868 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:29.877903 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 851ca41b87fa4f2e9794a407a3534868 (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:29.822634 7569 raft_consensus.cc:738] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
W20250626 19:58:29.886953 6703 catalog_manager.cc:4895] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 851ca41b87fa4f2e9794a407a3534868 because the tablet was not found. No further retry: Not found: Tablet not found: 851ca41b87fa4f2e9794a407a3534868
I20250626 19:58:29.887928 7559 tablet_replica.cc:331] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:29.889685 7559 raft_consensus.cc:2241] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:29.895210 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 2b9dfd7e091e4d0ea3a698a12a6be585 (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:29.895465 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 2b9dfd7e091e4d0ea3a698a12a6be585 (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:29.896271 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b9dfd7e091e4d0ea3a698a12a6be585. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:29.896384 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b9dfd7e091e4d0ea3a698a12a6be585. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:29.886260 7568 raft_consensus.cc:357] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.908639 7568 raft_consensus.cc:383] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:29.908947 7568 raft_consensus.cc:738] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:29.909608 7559 raft_consensus.cc:2270] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:29.909623 7568 consensus_queue.cc:260] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.887513 7569 consensus_queue.cc:260] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:29.893935 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 2b9dfd7e091e4d0ea3a698a12a6be585 (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:29.913908 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b9dfd7e091e4d0ea3a698a12a6be585. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:29.915539 7559 ts_tablet_manager.cc:1905] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:29.925748 7569 ts_tablet_manager.cc:1428] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.106s user 0.005s sys 0.000s
I20250626 19:58:29.931427 7568 ts_tablet_manager.cc:1428] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.110s user 0.005s sys 0.004s
I20250626 19:58:29.934495 7473 raft_consensus.cc:2241] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:29.937096 7473 pending_rounds.cc:70] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7: Trying to abort 1 pending ops.
I20250626 19:58:29.937467 7473 pending_rounds.cc:77] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964558877552640 op_type: NO_OP noop_request { }
I20250626 19:58:29.937932 7473 raft_consensus.cc:2887] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:29.938339 7473 raft_consensus.cc:2270] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:29.946012 7554 consensus_queue.cc:1035] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:29.948520 7473 ts_tablet_manager.cc:1905] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:29.886600 7560 ts_tablet_manager.cc:1918] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:29.970588 7560 log.cc:1199] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/851ca41b87fa4f2e9794a407a3534868
I20250626 19:58:29.975359 7568 tablet_bootstrap.cc:492] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:29.996358 7568 tablet_bootstrap.cc:654] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:29.997977 7584 consensus_queue.cc:1035] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:30.016628 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet 052c389ae05f47649465c86991795f7f from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:29.971482 7560 ts_tablet_manager.cc:1939] T 851ca41b87fa4f2e9794a407a3534868 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:30.027029 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 851ca41b87fa4f2e9794a407a3534868 (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:30.031737 7560 tablet_replica.cc:331] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:30.032557 7560 raft_consensus.cc:2241] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:30.032713 7568 tablet_bootstrap.cc:492] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:30.033028 7560 raft_consensus.cc:2270] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.033097 7568 ts_tablet_manager.cc:1397] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.001s
I20250626 19:58:30.035569 7568 raft_consensus.cc:357] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.036185 7568 raft_consensus.cc:383] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:30.036440 7568 raft_consensus.cc:738] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:30.037034 7568 consensus_queue.cc:260] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.043133 7569 tablet_bootstrap.cc:492] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:30.045830 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 052c389ae05f47649465c86991795f7f
I20250626 19:58:30.047241 6711 catalog_manager.cc:6652] ReplaceTablet: tablet 052c389ae05f47649465c86991795f7f deleted and replaced by tablet 0e058e24605e4bbfa67d9fd254e90749
I20250626 19:58:30.048600 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 052c389ae05f47649465c86991795f7f with delete_type TABLET_DATA_DELETED (replaced by tablet 0e058e24605e4bbfa67d9fd254e90749) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:30.048799 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 052c389ae05f47649465c86991795f7f with delete_type TABLET_DATA_DELETED (replaced by tablet 0e058e24605e4bbfa67d9fd254e90749) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:30.049042 7569 tablet_bootstrap.cc:654] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:30.049557 7473 ts_tablet_manager.cc:1918] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:30.050311 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 052c389ae05f47649465c86991795f7f with delete_type TABLET_DATA_DELETED (replaced by tablet 0e058e24605e4bbfa67d9fd254e90749) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:30.050072 7473 log.cc:1199] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/180d9cb0d2874a8e9ad6d45fca6cb577
I20250626 19:58:30.051441 7467 tablet_bootstrap.cc:492] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:30.051571 7473 ts_tablet_manager.cc:1939] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:30.053828 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:30.055850 7559 ts_tablet_manager.cc:1918] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:30.056277 7559 log.cc:1199] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/180d9cb0d2874a8e9ad6d45fca6cb577
I20250626 19:58:30.056717 7467 tablet_bootstrap.cc:654] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:30.056941 7559 ts_tablet_manager.cc:1939] T 180d9cb0d2874a8e9ad6d45fca6cb577 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:30.060902 7568 ts_tablet_manager.cc:1428] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.027s user 0.002s sys 0.003s
I20250626 19:58:30.068583 6709 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 2b9dfd7e091e4d0ea3a698a12a6be585 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:30.071161 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet 8cce294daa0642aebaf570ad84cfe3c9 from {username='slave'} at 127.0.0.1:42822
W20250626 19:58:30.071702 6709 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 2b9dfd7e091e4d0ea3a698a12a6be585 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 2b9dfd7e091e4d0ea3a698a12a6be585 consensus state has no leader
I20250626 19:58:30.072271 7560 ts_tablet_manager.cc:1905] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.072963 7473 tablet_replica.cc:331] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:30.073700 7473 raft_consensus.cc:2241] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:30.076257 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:30.077133 7473 raft_consensus.cc:2270] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.077589 6709 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 2b9dfd7e091e4d0ea3a698a12a6be585 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:30.082353 7473 ts_tablet_manager.cc:1905] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.082996 7559 tablet_replica.cc:331] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:30.083853 7559 raft_consensus.cc:2241] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:30.089833 7559 raft_consensus.cc:2270] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.093796 7559 ts_tablet_manager.cc:1905] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.098806 7467 tablet_bootstrap.cc:492] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:30.099244 7467 ts_tablet_manager.cc:1397] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.000s
I20250626 19:58:30.101751 7467 raft_consensus.cc:357] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.102356 7467 raft_consensus.cc:383] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:30.102569 7467 raft_consensus.cc:738] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:30.103433 7467 consensus_queue.cc:260] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.106964 7467 ts_tablet_manager.cc:1428] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250626 19:58:30.111958 7569 tablet_bootstrap.cc:492] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:30.112493 7569 ts_tablet_manager.cc:1397] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.070s user 0.014s sys 0.004s
I20250626 19:58:30.115089 7569 raft_consensus.cc:357] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.120879 7560 ts_tablet_manager.cc:1918] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:30.121344 7569 raft_consensus.cc:383] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:30.121805 7560 log.cc:1199] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/180d9cb0d2874a8e9ad6d45fca6cb577
I20250626 19:58:30.122165 7569 raft_consensus.cc:738] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:30.123001 7560 ts_tablet_manager.cc:1939] T 180d9cb0d2874a8e9ad6d45fca6cb577 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:30.123734 7569 consensus_queue.cc:260] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.129704 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:30.141631 7569 ts_tablet_manager.cc:1428] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.029s user 0.004s sys 0.002s
I20250626 19:58:30.143633 7560 tablet_replica.cc:331] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:30.155442 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 8cce294daa0642aebaf570ad84cfe3c9
I20250626 19:58:30.155546 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 8cce294daa0642aebaf570ad84cfe3c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 95d18eeed63647e99f6b596d1d7387ab) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:30.157424 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 8cce294daa0642aebaf570ad84cfe3c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 95d18eeed63647e99f6b596d1d7387ab) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:30.158162 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 8cce294daa0642aebaf570ad84cfe3c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 95d18eeed63647e99f6b596d1d7387ab) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:30.156643 7560 raft_consensus.cc:2241] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:30.159178 7560 raft_consensus.cc:2270] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.161381 7560 ts_tablet_manager.cc:1905] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.167305 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 8cce294daa0642aebaf570ad84cfe3c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 95d18eeed63647e99f6b596d1d7387ab) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:30.168241 6711 catalog_manager.cc:6652] ReplaceTablet: tablet 8cce294daa0642aebaf570ad84cfe3c9 deleted and replaced by tablet 95d18eeed63647e99f6b596d1d7387ab
I20250626 19:58:30.169797 7473 ts_tablet_manager.cc:1918] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:30.170255 7473 log.cc:1199] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/052c389ae05f47649465c86991795f7f
I20250626 19:58:30.171064 7473 ts_tablet_manager.cc:1939] T 052c389ae05f47649465c86991795f7f P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:30.181365 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 052c389ae05f47649465c86991795f7f (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:30.184653 7559 ts_tablet_manager.cc:1918] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:30.187770 7473 tablet_replica.cc:331] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:30.194871 7473 raft_consensus.cc:2241] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:30.200143 7560 ts_tablet_manager.cc:1918] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:30.200847 7560 log.cc:1199] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/052c389ae05f47649465c86991795f7f
I20250626 19:58:30.201792 7559 log.cc:1199] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/052c389ae05f47649465c86991795f7f
I20250626 19:58:30.202437 7473 raft_consensus.cc:2270] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.203289 7560 ts_tablet_manager.cc:1939] T 052c389ae05f47649465c86991795f7f P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:30.205873 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 052c389ae05f47649465c86991795f7f (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:30.208401 7560 tablet_replica.cc:331] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:30.209252 7560 raft_consensus.cc:2241] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:30.210299 7560 raft_consensus.cc:2270] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.220968 7473 ts_tablet_manager.cc:1905] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.209856 7559 ts_tablet_manager.cc:1939] T 052c389ae05f47649465c86991795f7f P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:30.225456 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 052c389ae05f47649465c86991795f7f (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:30.228278 7560 ts_tablet_manager.cc:1905] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.229596 7559 tablet_replica.cc:331] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:30.230420 7559 raft_consensus.cc:2241] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:30.232273 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 0e058e24605e4bbfa67d9fd254e90749 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.233232 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e058e24605e4bbfa67d9fd254e90749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.235455 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 0e058e24605e4bbfa67d9fd254e90749 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.235455 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 0e058e24605e4bbfa67d9fd254e90749 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.236439 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e058e24605e4bbfa67d9fd254e90749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.236442 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e058e24605e4bbfa67d9fd254e90749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.255059 7559 raft_consensus.cc:2270] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.263314 7559 ts_tablet_manager.cc:1905] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.266487 6711 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 2b9dfd7e091e4d0ea3a698a12a6be585 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:30.268414 6709 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 2b9dfd7e091e4d0ea3a698a12a6be585 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:30.271872 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet 734d9d3331b44fc49e1f64b8caf1e4d8 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:30.272183 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 95d18eeed63647e99f6b596d1d7387ab (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.273150 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d18eeed63647e99f6b596d1d7387ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.276023 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 95d18eeed63647e99f6b596d1d7387ab (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.277144 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d18eeed63647e99f6b596d1d7387ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.270700 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 95d18eeed63647e99f6b596d1d7387ab (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.279628 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d18eeed63647e99f6b596d1d7387ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.279994 6711 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 2b9dfd7e091e4d0ea3a698a12a6be585 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:30.283186 7467 tablet_bootstrap.cc:492] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:30.286679 7560 ts_tablet_manager.cc:1918] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:30.287149 7560 log.cc:1199] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/8cce294daa0642aebaf570ad84cfe3c9
I20250626 19:58:30.287882 7560 ts_tablet_manager.cc:1939] T 8cce294daa0642aebaf570ad84cfe3c9 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:30.290449 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 8cce294daa0642aebaf570ad84cfe3c9 (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:30.292781 7467 tablet_bootstrap.cc:654] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:30.292968 7569 tablet_bootstrap.cc:492] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:30.301318 7569 tablet_bootstrap.cc:654] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:30.305052 7568 tablet_bootstrap.cc:492] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:30.311304 7568 tablet_bootstrap.cc:654] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:30.323890 7473 ts_tablet_manager.cc:1918] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:30.324301 7473 log.cc:1199] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/8cce294daa0642aebaf570ad84cfe3c9
I20250626 19:58:30.325022 7473 ts_tablet_manager.cc:1939] T 8cce294daa0642aebaf570ad84cfe3c9 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:30.332278 7467 tablet_bootstrap.cc:492] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:30.332831 7467 ts_tablet_manager.cc:1397] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.000s
I20250626 19:58:30.333365 7568 tablet_bootstrap.cc:492] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:30.333818 7568 ts_tablet_manager.cc:1397] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.004s
I20250626 19:58:30.335518 7467 raft_consensus.cc:357] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.336316 7568 raft_consensus.cc:357] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.336966 7568 raft_consensus.cc:383] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:30.339169 7569 tablet_bootstrap.cc:492] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:30.339661 7569 ts_tablet_manager.cc:1397] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.006s
I20250626 19:58:30.336690 7467 raft_consensus.cc:383] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:30.342223 7569 raft_consensus.cc:357] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.346982 7467 raft_consensus.cc:738] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:30.347071 7569 raft_consensus.cc:383] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:30.347329 7569 raft_consensus.cc:738] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:30.347698 7568 raft_consensus.cc:738] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:30.348838 7568 consensus_queue.cc:260] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.349797 7569 consensus_queue.cc:260] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.351353 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 8cce294daa0642aebaf570ad84cfe3c9 (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:30.351135 7467 consensus_queue.cc:260] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.353945 7568 ts_tablet_manager.cc:1428] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20250626 19:58:30.356771 7467 ts_tablet_manager.cc:1428] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20250626 19:58:30.366593 7467 tablet_bootstrap.cc:492] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
W20250626 19:58:30.368628 6705 catalog_manager.cc:4895] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 8cce294daa0642aebaf570ad84cfe3c9 because the tablet was not found. No further retry: Not found: Tablet not found: 8cce294daa0642aebaf570ad84cfe3c9
I20250626 19:58:30.376976 7569 ts_tablet_manager.cc:1428] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.037s user 0.004s sys 0.002s
I20250626 19:58:30.382355 7467 tablet_bootstrap.cc:654] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:30.382602 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 734d9d3331b44fc49e1f64b8caf1e4d8 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:30.392278 7568 tablet_bootstrap.cc:492] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:30.377882 7559 ts_tablet_manager.cc:1918] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:30.392881 7559 log.cc:1199] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/8cce294daa0642aebaf570ad84cfe3c9
I20250626 19:58:30.399305 7559 ts_tablet_manager.cc:1939] T 8cce294daa0642aebaf570ad84cfe3c9 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:30.405340 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 8cce294daa0642aebaf570ad84cfe3c9 (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:30.398375 7569 tablet_bootstrap.cc:492] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:30.413369 7467 tablet_bootstrap.cc:492] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:30.413815 7467 ts_tablet_manager.cc:1397] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.000s
I20250626 19:58:30.416278 7467 raft_consensus.cc:357] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.416930 7467 raft_consensus.cc:383] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:30.417268 7568 tablet_bootstrap.cc:654] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:30.417196 7467 raft_consensus.cc:738] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:30.420333 7569 tablet_bootstrap.cc:654] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:30.423398 7467 consensus_queue.cc:260] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.432199 7568 tablet_bootstrap.cc:492] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:30.432730 7568 ts_tablet_manager.cc:1397] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.005s
I20250626 19:58:30.435721 7467 ts_tablet_manager.cc:1428] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20250626 19:58:30.457442 7569 tablet_bootstrap.cc:492] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:30.458007 7569 ts_tablet_manager.cc:1397] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.009s
I20250626 19:58:30.459652 7568 raft_consensus.cc:357] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.462122 7569 raft_consensus.cc:357] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.462885 7568 raft_consensus.cc:383] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:30.463246 7568 raft_consensus.cc:738] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:30.464051 7568 consensus_queue.cc:260] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.465104 7569 raft_consensus.cc:383] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:30.465435 7569 raft_consensus.cc:738] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:30.467020 7569 consensus_queue.cc:260] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.482334 7568 ts_tablet_manager.cc:1428] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20250626 19:58:30.489686 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet 0e058e24605e4bbfa67d9fd254e90749 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:30.493283 7569 ts_tablet_manager.cc:1428] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20250626 19:58:30.486900 7584 raft_consensus.cc:491] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:30.493813 7584 raft_consensus.cc:513] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.495721 7584 leader_election.cc:290] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:30.504956 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e058e24605e4bbfa67d9fd254e90749" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:30.505558 7360 raft_consensus.cc:2466] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:30.506966 7152 leader_election.cc:304] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:30.509088 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e058e24605e4bbfa67d9fd254e90749" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:30.509647 7073 raft_consensus.cc:2466] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:30.511515 7584 raft_consensus.cc:2802] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:30.511940 7584 raft_consensus.cc:491] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:30.512333 7584 raft_consensus.cc:3058] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:30.527838 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 0e058e24605e4bbfa67d9fd254e90749
I20250626 19:58:30.529541 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e058e24605e4bbfa67d9fd254e90749 with delete_type TABLET_DATA_DELETED (replaced by tablet eb3156e127514c95b175a87c015d0dac) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:30.530470 7559 tablet_replica.cc:331] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:30.531307 7559 raft_consensus.cc:2241] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:30.531829 7559 raft_consensus.cc:2270] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.534267 7559 ts_tablet_manager.cc:1905] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.538955 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e058e24605e4bbfa67d9fd254e90749 with delete_type TABLET_DATA_DELETED (replaced by tablet eb3156e127514c95b175a87c015d0dac) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:30.539795 7473 tablet_replica.cc:331] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:30.540551 7473 raft_consensus.cc:2241] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:30.541033 7473 raft_consensus.cc:2270] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.541997 6711 catalog_manager.cc:6652] ReplaceTablet: tablet 0e058e24605e4bbfa67d9fd254e90749 deleted and replaced by tablet eb3156e127514c95b175a87c015d0dac
I20250626 19:58:30.543354 7473 ts_tablet_manager.cc:1905] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.544633 7573 raft_consensus.cc:491] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:30.545094 7573 raft_consensus.cc:513] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.546974 7573 leader_election.cc:290] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:30.548859 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d18eeed63647e99f6b596d1d7387ab" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:30.549458 7360 raft_consensus.cc:2466] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:30.550776 7009 leader_election.cc:304] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:30.551630 7573 raft_consensus.cc:2802] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:30.551961 7573 raft_consensus.cc:491] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:30.552251 7573 raft_consensus.cc:3058] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:30.561360 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e058e24605e4bbfa67d9fd254e90749 with delete_type TABLET_DATA_DELETED (replaced by tablet eb3156e127514c95b175a87c015d0dac) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:30.562098 7560 tablet_replica.cc:331] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:30.563437 7475 raft_consensus.cc:491] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:30.563515 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d18eeed63647e99f6b596d1d7387ab" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:30.564118 7215 raft_consensus.cc:2466] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:30.563901 7475 raft_consensus.cc:513] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.565937 7475 leader_election.cc:290] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:30.567677 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b9dfd7e091e4d0ea3a698a12a6be585" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:30.568370 7073 raft_consensus.cc:2466] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:30.565449 7573 raft_consensus.cc:513] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.570287 7150 leader_election.cc:304] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:30.572113 7475 raft_consensus.cc:2802] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:30.574152 7475 raft_consensus.cc:491] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:30.574574 7475 raft_consensus.cc:3058] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:30.576481 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b9dfd7e091e4d0ea3a698a12a6be585" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:30.573680 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d18eeed63647e99f6b596d1d7387ab" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:30.577126 7215 raft_consensus.cc:3058] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:30.577102 7360 raft_consensus.cc:2466] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:30.595724 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 95d18eeed63647e99f6b596d1d7387ab from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:30.597885 7559 ts_tablet_manager.cc:1918] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:30.598402 7559 log.cc:1199] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/0e058e24605e4bbfa67d9fd254e90749
I20250626 19:58:30.599357 7215 raft_consensus.cc:2466] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:30.601323 7051 tablet_service.cc:1468] Processing CreateTablet for tablet eb3156e127514c95b175a87c015d0dac (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.602381 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3156e127514c95b175a87c015d0dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.605458 7006 leader_election.cc:304] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:30.609251 7584 raft_consensus.cc:513] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.613937 7473 ts_tablet_manager.cc:1918] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:30.614351 7473 log.cc:1199] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/0e058e24605e4bbfa67d9fd254e90749
I20250626 19:58:30.613971 7194 tablet_service.cc:1468] Processing CreateTablet for tablet eb3156e127514c95b175a87c015d0dac (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.615006 7473 ts_tablet_manager.cc:1939] T 0e058e24605e4bbfa67d9fd254e90749 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:30.615746 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3156e127514c95b175a87c015d0dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.615564 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d18eeed63647e99f6b596d1d7387ab" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:30.616077 7360 raft_consensus.cc:3058] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:30.618238 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 0e058e24605e4bbfa67d9fd254e90749 (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:30.623190 7360 raft_consensus.cc:2466] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:30.623322 7559 ts_tablet_manager.cc:1939] T 0e058e24605e4bbfa67d9fd254e90749 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:30.624411 7573 leader_election.cc:290] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Requested vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:30.603996 7475 raft_consensus.cc:513] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.624931 7573 raft_consensus.cc:2802] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:30.625357 7573 raft_consensus.cc:695] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Becoming Leader. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Running, Role: LEADER
I20250626 19:58:30.626011 7560 raft_consensus.cc:2241] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:30.626389 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 0e058e24605e4bbfa67d9fd254e90749 (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:30.628237 7560 raft_consensus.cc:2270] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.636757 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b9dfd7e091e4d0ea3a698a12a6be585" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:30.637331 7361 raft_consensus.cc:3058] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:30.646195 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 95d18eeed63647e99f6b596d1d7387ab
I20250626 19:58:30.646440 7361 raft_consensus.cc:2466] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:30.647763 7152 leader_election.cc:304] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:30.647915 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 95d18eeed63647e99f6b596d1d7387ab with delete_type TABLET_DATA_DELETED (replaced by tablet 131f263d07394c0d90edcc5637e3af51) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:30.649478 7559 tablet_replica.cc:331] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:30.653080 7339 tablet_service.cc:1468] Processing CreateTablet for tablet eb3156e127514c95b175a87c015d0dac (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.654345 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3156e127514c95b175a87c015d0dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.657151 7609 raft_consensus.cc:2802] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
W20250626 19:58:30.659761 7152 leader_election.cc:343] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: 0e058e24605e4bbfa67d9fd254e90749
I20250626 19:58:30.660146 7475 leader_election.cc:290] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:30.660447 7584 leader_election.cc:290] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:30.664192 7560 ts_tablet_manager.cc:1905] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.670276 7568 tablet_bootstrap.cc:492] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:30.676162 7568 tablet_bootstrap.cc:654] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:30.679162 7573 consensus_queue.cc:237] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.684062 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 95d18eeed63647e99f6b596d1d7387ab with delete_type TABLET_DATA_DELETED (replaced by tablet 131f263d07394c0d90edcc5637e3af51) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:30.684345 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 95d18eeed63647e99f6b596d1d7387ab with delete_type TABLET_DATA_DELETED (replaced by tablet 131f263d07394c0d90edcc5637e3af51) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:30.685106 7473 tablet_replica.cc:331] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:30.685957 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 95d18eeed63647e99f6b596d1d7387ab deleted and replaced by tablet 131f263d07394c0d90edcc5637e3af51
I20250626 19:58:30.686124 7473 raft_consensus.cc:2241] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:30.686726 7473 raft_consensus.cc:2270] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.689877 7473 ts_tablet_manager.cc:1905] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:58:30.689913 7006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:40373, user_credentials={real_user=slave}} blocked reactor thread for 84633.7us
I20250626 19:58:30.691407 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b9dfd7e091e4d0ea3a698a12a6be585" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:30.691944 7073 raft_consensus.cc:3058] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:30.697602 7568 tablet_bootstrap.cc:492] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:30.698069 7568 ts_tablet_manager.cc:1397] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.000s
I20250626 19:58:30.698518 7073 raft_consensus.cc:2466] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:30.700619 7568 raft_consensus.cc:357] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.701290 7568 raft_consensus.cc:383] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250626 19:58:30.701534 7150 leader_election.cc:343] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Not found: Tablet not found: 0e058e24605e4bbfa67d9fd254e90749
I20250626 19:58:30.701946 7150 leader_election.cc:304] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a; no voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca
W20250626 19:58:30.702508 7150 raft_consensus.cc:2684] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250626 19:58:30.708798 7609 raft_consensus.cc:695] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:30.711015 7609 consensus_queue.cc:237] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.716187 7559 raft_consensus.cc:2241] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:30.701565 7568 raft_consensus.cc:738] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:30.716744 7467 tablet_bootstrap.cc:492] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:30.718712 7568 consensus_queue.cc:260] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.722846 7568 ts_tablet_manager.cc:1428] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20250626 19:58:30.723349 7467 tablet_bootstrap.cc:654] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:30.751125 7467 tablet_bootstrap.cc:492] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:30.750233 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 131f263d07394c0d90edcc5637e3af51 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.751500 7467 ts_tablet_manager.cc:1397] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.000s
I20250626 19:58:30.753973 7467 raft_consensus.cc:357] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.754556 7467 raft_consensus.cc:383] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:30.754508 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 131f263d07394c0d90edcc5637e3af51 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.754895 7467 raft_consensus.cc:738] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:30.755349 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 131f263d07394c0d90edcc5637e3af51 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.755620 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 131f263d07394c0d90edcc5637e3af51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.756349 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 131f263d07394c0d90edcc5637e3af51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.767104 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 131f263d07394c0d90edcc5637e3af51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.769506 7473 ts_tablet_manager.cc:1918] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:30.769923 7473 log.cc:1199] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/95d18eeed63647e99f6b596d1d7387ab
I20250626 19:58:30.770529 7473 ts_tablet_manager.cc:1939] T 95d18eeed63647e99f6b596d1d7387ab P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:30.771867 7559 pending_rounds.cc:70] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca: Trying to abort 1 pending ops.
I20250626 19:58:30.772133 7559 pending_rounds.cc:77] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964562214998016 op_type: NO_OP noop_request { }
I20250626 19:58:30.772508 7559 raft_consensus.cc:2887] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:30.772569 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 2b9dfd7e091e4d0ea3a698a12a6be585 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:30.772835 7559 raft_consensus.cc:2270] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.773993 7569 tablet_bootstrap.cc:492] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:30.779995 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 95d18eeed63647e99f6b596d1d7387ab (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
W20250626 19:58:30.781886 7612 consensus_queue.cc:1162] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "bbe151be01e24c82854d9c8c79c797ca" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250626 19:58:30.788532 7568 tablet_bootstrap.cc:492] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:30.794373 7568 tablet_bootstrap.cc:654] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:30.799422 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 2b9dfd7e091e4d0ea3a698a12a6be585
I20250626 19:58:30.800676 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 2b9dfd7e091e4d0ea3a698a12a6be585 deleted and replaced by tablet 87b3b241727348a19c66c4ffcb0a675e
I20250626 19:58:30.802587 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b9dfd7e091e4d0ea3a698a12a6be585 with delete_type TABLET_DATA_DELETED (replaced by tablet 87b3b241727348a19c66c4ffcb0a675e) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:30.803261 7473 tablet_replica.cc:331] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:30.803912 7473 raft_consensus.cc:2241] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:30.804337 7473 raft_consensus.cc:2270] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.805924 7473 ts_tablet_manager.cc:1905] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.806154 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b9dfd7e091e4d0ea3a698a12a6be585 with delete_type TABLET_DATA_DELETED (replaced by tablet 87b3b241727348a19c66c4ffcb0a675e) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:30.809394 7559 ts_tablet_manager.cc:1905] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.809701 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b9dfd7e091e4d0ea3a698a12a6be585 with delete_type TABLET_DATA_DELETED (replaced by tablet 87b3b241727348a19c66c4ffcb0a675e) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:30.810583 7467 consensus_queue.cc:260] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.812713 7467 ts_tablet_manager.cc:1428] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.061s user 0.005s sys 0.000s
I20250626 19:58:30.813604 7467 tablet_bootstrap.cc:492] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:30.816701 7560 ts_tablet_manager.cc:1918] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:30.817087 7560 log.cc:1199] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/0e058e24605e4bbfa67d9fd254e90749
I20250626 19:58:30.817685 7560 ts_tablet_manager.cc:1939] T 0e058e24605e4bbfa67d9fd254e90749 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:30.825641 7568 tablet_bootstrap.cc:492] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:30.826192 7568 ts_tablet_manager.cc:1397] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.000s
I20250626 19:58:30.848578 7473 ts_tablet_manager.cc:1918] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:30.849025 7473 log.cc:1199] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/2b9dfd7e091e4d0ea3a698a12a6be585
I20250626 19:58:30.849627 7473 ts_tablet_manager.cc:1939] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:30.851764 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 2b9dfd7e091e4d0ea3a698a12a6be585 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:30.852675 7568 raft_consensus.cc:357] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.853292 7568 raft_consensus.cc:383] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:30.853524 7568 raft_consensus.cc:738] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:30.854141 7568 consensus_queue.cc:260] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.860286 7568 ts_tablet_manager.cc:1428] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20250626 19:58:30.863858 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 87b3b241727348a19c66c4ffcb0a675e (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.864951 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87b3b241727348a19c66c4ffcb0a675e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.866436 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 87b3b241727348a19c66c4ffcb0a675e (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.867345 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87b3b241727348a19c66c4ffcb0a675e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.868121 7560 tablet_replica.cc:331] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:30.868809 7560 raft_consensus.cc:2241] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:30.869313 7560 raft_consensus.cc:2270] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 19:58:30.870288 6704 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 180d9cb0d2874a8e9ad6d45fca6cb577 consensus state has no leader
I20250626 19:58:30.871783 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 0e058e24605e4bbfa67d9fd254e90749 (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:30.888099 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 87b3b241727348a19c66c4ffcb0a675e (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:30.889189 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87b3b241727348a19c66c4ffcb0a675e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:30.898394 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet eb3156e127514c95b175a87c015d0dac from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:30.907953 7559 ts_tablet_manager.cc:1918] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:30.908347 7559 log.cc:1199] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/95d18eeed63647e99f6b596d1d7387ab
I20250626 19:58:30.922618 7559 ts_tablet_manager.cc:1939] T 95d18eeed63647e99f6b596d1d7387ab P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:30.925215 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 95d18eeed63647e99f6b596d1d7387ab (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:30.926433 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet eb3156e127514c95b175a87c015d0dac
I20250626 19:58:30.928805 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet eb3156e127514c95b175a87c015d0dac with delete_type TABLET_DATA_DELETED (replaced by tablet 9216bd5427f74c0dbf23c21240b2e17d) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:30.929575 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet eb3156e127514c95b175a87c015d0dac with delete_type TABLET_DATA_DELETED (replaced by tablet 9216bd5427f74c0dbf23c21240b2e17d) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:30.929493 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet eb3156e127514c95b175a87c015d0dac with delete_type TABLET_DATA_DELETED (replaced by tablet 9216bd5427f74c0dbf23c21240b2e17d) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:30.930318 7473 tablet_replica.cc:331] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:30.930965 7568 tablet_bootstrap.cc:492] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:30.931010 7473 raft_consensus.cc:2241] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:30.931409 7473 raft_consensus.cc:2270] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.933495 7473 ts_tablet_manager.cc:1905] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.935446 7569 tablet_bootstrap.cc:700] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca: Time spent opening tablet: real 0.109s user 0.003s sys 0.001s
I20250626 19:58:30.936120 7559 tablet_replica.cc:331] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:30.938493 6711 catalog_manager.cc:6652] ReplaceTablet: tablet eb3156e127514c95b175a87c015d0dac deleted and replaced by tablet 9216bd5427f74c0dbf23c21240b2e17d
I20250626 19:58:30.939684 7559 raft_consensus.cc:2241] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:30.940165 7559 raft_consensus.cc:2270] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:30.941768 7559 ts_tablet_manager.cc:1905] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.936172 7569 tablet_bootstrap.cc:654] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:30.961827 6704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250626 19:58:30.967043 7473 ts_tablet_manager.cc:1918] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:30.967473 7473 log.cc:1199] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/eb3156e127514c95b175a87c015d0dac
I20250626 19:58:30.967957 7559 ts_tablet_manager.cc:1918] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:30.968128 7473 ts_tablet_manager.cc:1939] T eb3156e127514c95b175a87c015d0dac P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:30.968524 7559 log.cc:1199] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/2b9dfd7e091e4d0ea3a698a12a6be585
I20250626 19:58:30.969174 7559 ts_tablet_manager.cc:1939] T 2b9dfd7e091e4d0ea3a698a12a6be585 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:30.970427 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet eb3156e127514c95b175a87c015d0dac (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:30.973194 7560 ts_tablet_manager.cc:1905] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:30.976420 7569 tablet_bootstrap.cc:492] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:30.976824 7569 ts_tablet_manager.cc:1397] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.203s user 0.006s sys 0.007s
I20250626 19:58:30.979728 7467 tablet_bootstrap.cc:654] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:30.986208 7569 raft_consensus.cc:357] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:30.987053 7569 raft_consensus.cc:383] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:30.987401 7569 raft_consensus.cc:738] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:30.988233 7569 consensus_queue.cc:260] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:31.006003 6704 consensus_peers.cc:487] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f -> Peer 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417): Couldn't send request to peer 8ae38ec946e14198ae4adc8dde9d66e8. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250626 19:58:31.015413 7568 tablet_bootstrap.cc:654] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:31.019230 7569 ts_tablet_manager.cc:1428] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.042s user 0.006s sys 0.001s
I20250626 19:58:31.023908 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 131f263d07394c0d90edcc5637e3af51 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:31.027935 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 2b9dfd7e091e4d0ea3a698a12a6be585 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:31.028936 7559 tablet_replica.cc:331] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:31.030275 7559 raft_consensus.cc:2241] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:31.029588 7569 tablet_bootstrap.cc:492] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:31.033109 7559 raft_consensus.cc:2270] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:31.037422 7559 ts_tablet_manager.cc:1905] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:31.041496 7569 tablet_bootstrap.cc:654] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:31.049572 7560 ts_tablet_manager.cc:1918] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:31.056106 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 9216bd5427f74c0dbf23c21240b2e17d (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:31.057175 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9216bd5427f74c0dbf23c21240b2e17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:31.057457 7467 tablet_bootstrap.cc:492] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:31.060505 7467 ts_tablet_manager.cc:1397] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.247s user 0.011s sys 0.000s
I20250626 19:58:31.061676 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 131f263d07394c0d90edcc5637e3af51
I20250626 19:58:31.063428 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 131f263d07394c0d90edcc5637e3af51 deleted and replaced by tablet 30bf36fa65414141a2e633ce72a9d7c4
I20250626 19:58:31.065637 7341 tablet_service.cc:1515] Processing DeleteTablet for tablet 131f263d07394c0d90edcc5637e3af51 with delete_type TABLET_DATA_DELETED (replaced by tablet 30bf36fa65414141a2e633ce72a9d7c4) from {username='slave'} at 127.0.0.1:43274
W20250626 19:58:31.067315 6705 catalog_manager.cc:4908] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 131f263d07394c0d90edcc5637e3af51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 131f263d07394c0d90edcc5637e3af51 already in progress: creating tablet
I20250626 19:58:31.068226 7467 raft_consensus.cc:357] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.070086 7467 raft_consensus.cc:383] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.070325 7467 raft_consensus.cc:738] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.051337 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 9216bd5427f74c0dbf23c21240b2e17d (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:31.080619 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9216bd5427f74c0dbf23c21240b2e17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:31.082244 7559 ts_tablet_manager.cc:1918] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:31.082551 7559 log.cc:1199] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/eb3156e127514c95b175a87c015d0dac
I20250626 19:58:31.063231 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 131f263d07394c0d90edcc5637e3af51 with delete_type TABLET_DATA_DELETED (replaced by tablet 30bf36fa65414141a2e633ce72a9d7c4) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:31.073882 7467 consensus_queue.cc:260] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.091601 7467 ts_tablet_manager.cc:1428] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20250626 19:58:31.096282 7467 tablet_bootstrap.cc:492] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:31.054934 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 9216bd5427f74c0dbf23c21240b2e17d (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:31.098325 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9216bd5427f74c0dbf23c21240b2e17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:31.102098 7559 ts_tablet_manager.cc:1939] T eb3156e127514c95b175a87c015d0dac P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:31.064191 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 131f263d07394c0d90edcc5637e3af51 with delete_type TABLET_DATA_DELETED (replaced by tablet 30bf36fa65414141a2e633ce72a9d7c4) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:31.104233 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet eb3156e127514c95b175a87c015d0dac (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:31.094583 7568 tablet_bootstrap.cc:492] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:31.107285 7568 ts_tablet_manager.cc:1397] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.177s user 0.015s sys 0.002s
W20250626 19:58:31.108354 6703 catalog_manager.cc:4908] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 131f263d07394c0d90edcc5637e3af51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 131f263d07394c0d90edcc5637e3af51 already in progress: creating tablet
I20250626 19:58:31.114318 7568 raft_consensus.cc:357] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.117192 7568 raft_consensus.cc:383] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.117534 7568 raft_consensus.cc:738] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.115157 7560 log.cc:1199] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/95d18eeed63647e99f6b596d1d7387ab
I20250626 19:58:31.118690 7560 ts_tablet_manager.cc:1939] T 95d18eeed63647e99f6b596d1d7387ab P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:31.121101 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 95d18eeed63647e99f6b596d1d7387ab (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:31.121251 7568 consensus_queue.cc:260] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.127252 7467 tablet_bootstrap.cc:654] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:31.127806 7568 ts_tablet_manager.cc:1428] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.020s user 0.003s sys 0.002s
I20250626 19:58:31.140417 7560 tablet_replica.cc:331] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:31.141094 7560 raft_consensus.cc:2241] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:31.141726 7560 pending_rounds.cc:70] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a: Trying to abort 1 pending ops.
I20250626 19:58:31.141956 7560 pending_rounds.cc:77] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964562283065344 op_type: NO_OP noop_request { }
I20250626 19:58:31.142279 7560 raft_consensus.cc:2887] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:31.142602 7560 raft_consensus.cc:2270] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:31.142810 7467 tablet_bootstrap.cc:492] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:31.144434 7467 ts_tablet_manager.cc:1397] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.000s
I20250626 19:58:31.145539 7568 tablet_bootstrap.cc:492] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:31.146332 6711 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 87b3b241727348a19c66c4ffcb0a675e (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
W20250626 19:58:31.148334 6711 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 87b3b241727348a19c66c4ffcb0a675e (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 87b3b241727348a19c66c4ffcb0a675e consensus state has no leader
I20250626 19:58:31.146700 7467 raft_consensus.cc:357] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.149703 7467 raft_consensus.cc:383] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.149971 7467 raft_consensus.cc:738] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.153081 7467 consensus_queue.cc:260] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.158555 6711 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 87b3b241727348a19c66c4ffcb0a675e (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:31.161521 7568 tablet_bootstrap.cc:654] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:31.163314 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 9216bd5427f74c0dbf23c21240b2e17d from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:31.163676 7467 ts_tablet_manager.cc:1428] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250626 19:58:31.164580 7467 tablet_bootstrap.cc:492] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:31.166298 7569 tablet_bootstrap.cc:492] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:31.166723 7569 ts_tablet_manager.cc:1397] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.137s user 0.011s sys 0.000s
I20250626 19:58:31.169023 7569 raft_consensus.cc:357] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.169699 7569 raft_consensus.cc:383] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.169988 7569 raft_consensus.cc:738] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.171222 7560 ts_tablet_manager.cc:1905] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:31.176208 7569 consensus_queue.cc:260] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.186115 7569 ts_tablet_manager.cc:1428] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250626 19:58:31.189105 7467 tablet_bootstrap.cc:654] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:31.190615 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 9216bd5427f74c0dbf23c21240b2e17d
I20250626 19:58:31.191766 7569 tablet_bootstrap.cc:492] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:31.193106 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 9216bd5427f74c0dbf23c21240b2e17d with delete_type TABLET_DATA_DELETED (replaced by tablet b65e085b67684ca18c95730b8a818286) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:31.193958 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 9216bd5427f74c0dbf23c21240b2e17d with delete_type TABLET_DATA_DELETED (replaced by tablet b65e085b67684ca18c95730b8a818286) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:31.194307 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 9216bd5427f74c0dbf23c21240b2e17d deleted and replaced by tablet b65e085b67684ca18c95730b8a818286
I20250626 19:58:31.194716 7473 tablet_replica.cc:331] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:31.195417 7473 raft_consensus.cc:2241] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:31.195932 7473 raft_consensus.cc:2270] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:31.193929 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 9216bd5427f74c0dbf23c21240b2e17d with delete_type TABLET_DATA_DELETED (replaced by tablet b65e085b67684ca18c95730b8a818286) from {username='slave'} at 127.0.0.1:54698
W20250626 19:58:31.198668 6703 catalog_manager.cc:4908] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 9216bd5427f74c0dbf23c21240b2e17d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9216bd5427f74c0dbf23c21240b2e17d already in progress: creating tablet
I20250626 19:58:31.198967 7473 ts_tablet_manager.cc:1905] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:31.202159 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 131f263d07394c0d90edcc5637e3af51 with delete_type TABLET_DATA_DELETED (replaced by tablet 30bf36fa65414141a2e633ce72a9d7c4) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:31.217370 7467 tablet_bootstrap.cc:492] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:31.217767 7467 ts_tablet_manager.cc:1397] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.006s
I20250626 19:58:31.219913 7467 raft_consensus.cc:357] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.220520 7467 raft_consensus.cc:383] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.220767 7467 raft_consensus.cc:738] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.221637 7569 tablet_bootstrap.cc:654] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:31.226704 7568 tablet_bootstrap.cc:492] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:31.227145 7568 ts_tablet_manager.cc:1397] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.001s
I20250626 19:58:31.229596 7568 raft_consensus.cc:357] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.230232 7568 raft_consensus.cc:383] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.230487 7568 raft_consensus.cc:738] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.232136 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 131f263d07394c0d90edcc5637e3af51 with delete_type TABLET_DATA_DELETED (replaced by tablet 30bf36fa65414141a2e633ce72a9d7c4) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:31.232797 7559 tablet_replica.cc:331] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:31.233431 7559 raft_consensus.cc:2241] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:31.233850 7559 raft_consensus.cc:2270] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:31.230865 7467 consensus_queue.cc:260] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.236131 7559 ts_tablet_manager.cc:1905] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:31.243546 7568 consensus_queue.cc:260] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.248596 7467 ts_tablet_manager.cc:1428] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20250626 19:58:31.252058 7568 ts_tablet_manager.cc:1428] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.025s user 0.003s sys 0.002s
I20250626 19:58:31.258888 7473 ts_tablet_manager.cc:1918] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:31.259277 7473 log.cc:1199] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/9216bd5427f74c0dbf23c21240b2e17d
I20250626 19:58:31.259903 7473 ts_tablet_manager.cc:1939] T 9216bd5427f74c0dbf23c21240b2e17d P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:31.273523 7473 tablet_replica.cc:331] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:31.274212 7473 raft_consensus.cc:2241] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:31.274672 7473 raft_consensus.cc:2270] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:31.279088 7473 ts_tablet_manager.cc:1905] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:31.262297 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 9216bd5427f74c0dbf23c21240b2e17d (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:31.281246 7559 ts_tablet_manager.cc:1918] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:31.291005 7560 ts_tablet_manager.cc:1918] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:31.291498 7560 log.cc:1199] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/2b9dfd7e091e4d0ea3a698a12a6be585
I20250626 19:58:31.292263 7560 ts_tablet_manager.cc:1939] T 2b9dfd7e091e4d0ea3a698a12a6be585 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:31.293394 7473 ts_tablet_manager.cc:1918] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:31.284543 7569 tablet_bootstrap.cc:492] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:31.293730 7473 log.cc:1199] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/131f263d07394c0d90edcc5637e3af51
I20250626 19:58:31.294072 7569 ts_tablet_manager.cc:1397] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.103s user 0.009s sys 0.003s
I20250626 19:58:31.294315 7473 ts_tablet_manager.cc:1939] T 131f263d07394c0d90edcc5637e3af51 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:31.296375 7569 raft_consensus.cc:357] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.296942 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 131f263d07394c0d90edcc5637e3af51 (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:31.297019 7569 raft_consensus.cc:383] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.297446 7569 raft_consensus.cc:738] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.297204 7559 log.cc:1199] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/131f263d07394c0d90edcc5637e3af51
I20250626 19:58:31.299465 7559 ts_tablet_manager.cc:1939] T 131f263d07394c0d90edcc5637e3af51 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:31.298271 7569 consensus_queue.cc:260] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.309914 7569 ts_tablet_manager.cc:1428] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250626 19:58:31.311682 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 2b9dfd7e091e4d0ea3a698a12a6be585 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:31.320899 7569 tablet_bootstrap.cc:492] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:31.330085 7569 tablet_bootstrap.cc:654] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:31.330583 7560 tablet_replica.cc:331] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:31.332655 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 131f263d07394c0d90edcc5637e3af51 (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:31.337025 7560 raft_consensus.cc:2241] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:31.340293 7560 raft_consensus.cc:2270] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:31.342049 7560 ts_tablet_manager.cc:1905] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:31.365679 6709 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 87b3b241727348a19c66c4ffcb0a675e (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:31.367347 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 30bf36fa65414141a2e633ce72a9d7c4 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:31.367702 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 1d7a74392e9d475588838a811011e9d6 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:31.368384 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30bf36fa65414141a2e633ce72a9d7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:31.390498 7609 raft_consensus.cc:491] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:31.391088 7609 raft_consensus.cc:513] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.392899 7609 leader_election.cc:290] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:31.394594 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 1d7a74392e9d475588838a811011e9d6
I20250626 19:58:31.394793 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9216bd5427f74c0dbf23c21240b2e17d" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:31.395948 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 1d7a74392e9d475588838a811011e9d6 deleted and replaced by tablet 973b4134faa440469635b796847674ab
W20250626 19:58:31.396148 7152 leader_election.cc:343] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: 9216bd5427f74c0dbf23c21240b2e17d
I20250626 19:58:31.398190 6709 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 87b3b241727348a19c66c4ffcb0a675e (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
W20250626 19:58:31.401345 7150 leader_election.cc:343] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:31.401895 7150 leader_election.cc:304] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a; no voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:31.403321 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d7a74392e9d475588838a811011e9d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 973b4134faa440469635b796847674ab) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:31.404421 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d7a74392e9d475588838a811011e9d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 973b4134faa440469635b796847674ab) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:31.405104 7559 tablet_replica.cc:331] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:31.405745 7559 raft_consensus.cc:2241] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:31.406190 7559 raft_consensus.cc:2270] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:31.408283 7559 ts_tablet_manager.cc:1905] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:31.412571 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 1d7a74392e9d475588838a811011e9d6 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:31.413295 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d7a74392e9d475588838a811011e9d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 973b4134faa440469635b796847674ab) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:31.413990 7473 tablet_replica.cc:331] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:31.414592 7473 raft_consensus.cc:2241] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:31.415613 7473 raft_consensus.cc:2270] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:31.417829 7473 ts_tablet_manager.cc:1905] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:31.424515 7609 raft_consensus.cc:2747] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:31.437448 7560 ts_tablet_manager.cc:1918] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:31.443209 7560 log.cc:1199] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/eb3156e127514c95b175a87c015d0dac
I20250626 19:58:31.444664 7560 ts_tablet_manager.cc:1939] T eb3156e127514c95b175a87c015d0dac P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:31.453540 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet eb3156e127514c95b175a87c015d0dac (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:31.457415 7560 tablet_replica.cc:331] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:31.459599 7560 raft_consensus.cc:2241] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:31.462354 7560 raft_consensus.cc:2270] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:31.467728 7560 ts_tablet_manager.cc:1905] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:31.481081 6711 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 87b3b241727348a19c66c4ffcb0a675e (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:31.486763 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 30bf36fa65414141a2e633ce72a9d7c4 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:31.487929 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30bf36fa65414141a2e633ce72a9d7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:31.488418 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 30bf36fa65414141a2e633ce72a9d7c4 (DEFAULT_TABLE table=default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:31.489437 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30bf36fa65414141a2e633ce72a9d7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:31.491950 6711 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 87b3b241727348a19c66c4ffcb0a675e (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:31.509395 7052 tablet_service.cc:1468] Processing CreateTablet for tablet b65e085b67684ca18c95730b8a818286 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:31.510511 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b65e085b67684ca18c95730b8a818286. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:31.510731 7338 tablet_service.cc:1468] Processing CreateTablet for tablet b65e085b67684ca18c95730b8a818286 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:31.511653 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b65e085b67684ca18c95730b8a818286. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:31.516963 7196 tablet_service.cc:1468] Processing CreateTablet for tablet b65e085b67684ca18c95730b8a818286 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:31.518033 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b65e085b67684ca18c95730b8a818286. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:31.523432 7569 tablet_bootstrap.cc:492] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:31.524060 7569 ts_tablet_manager.cc:1397] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.203s user 0.007s sys 0.021s
I20250626 19:58:31.528653 7473 ts_tablet_manager.cc:1918] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:31.529048 7473 log.cc:1199] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/1d7a74392e9d475588838a811011e9d6
I20250626 19:58:31.529713 7473 ts_tablet_manager.cc:1939] T 1d7a74392e9d475588838a811011e9d6 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:31.532720 7559 ts_tablet_manager.cc:1918] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:31.533053 7559 log.cc:1199] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/1d7a74392e9d475588838a811011e9d6
I20250626 19:58:31.533880 7559 ts_tablet_manager.cc:1939] T 1d7a74392e9d475588838a811011e9d6 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:31.536315 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 1d7a74392e9d475588838a811011e9d6 (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
I20250626 19:58:31.540170 7467 tablet_bootstrap.cc:492] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:31.547454 7467 tablet_bootstrap.cc:654] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:31.526651 7569 raft_consensus.cc:357] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.551568 7569 raft_consensus.cc:383] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.551908 7569 raft_consensus.cc:738] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.552687 7569 consensus_queue.cc:260] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.559587 7467 tablet_bootstrap.cc:492] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:31.559980 7467 ts_tablet_manager.cc:1397] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
W20250626 19:58:31.565414 6711 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
I20250626 19:58:31.569397 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 1d7a74392e9d475588838a811011e9d6 (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
I20250626 19:58:31.570681 7467 raft_consensus.cc:357] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.571354 7467 raft_consensus.cc:383] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.571622 7467 raft_consensus.cc:738] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.572402 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 9216bd5427f74c0dbf23c21240b2e17d with delete_type TABLET_DATA_DELETED (replaced by tablet b65e085b67684ca18c95730b8a818286) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:31.572247 7467 consensus_queue.cc:260] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.577865 7568 tablet_bootstrap.cc:492] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
W20250626 19:58:31.579272 6703 catalog_manager.cc:4908] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 9216bd5427f74c0dbf23c21240b2e17d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9216bd5427f74c0dbf23c21240b2e17d already in progress: creating tablet
I20250626 19:58:31.586186 7568 tablet_bootstrap.cc:654] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:31.587519 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 973b4134faa440469635b796847674ab (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:31.588068 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 973b4134faa440469635b796847674ab (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:31.588667 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 973b4134faa440469635b796847674ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:31.589044 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 973b4134faa440469635b796847674ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:31.589859 7467 ts_tablet_manager.cc:1428] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250626 19:58:31.592443 7193 tablet_service.cc:1468] Processing CreateTablet for tablet 973b4134faa440469635b796847674ab (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:31.593566 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 973b4134faa440469635b796847674ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:31.599372 7467 tablet_bootstrap.cc:492] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:31.607064 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet ae04997b77924e8ba479c5c34ac822d7 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:31.612111 7569 ts_tablet_manager.cc:1428] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.088s user 0.005s sys 0.000s
I20250626 19:58:31.615074 7467 tablet_bootstrap.cc:654] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:31.626587 7467 tablet_bootstrap.cc:492] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:31.627004 7467 ts_tablet_manager.cc:1397] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.002s
I20250626 19:58:31.628012 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 9216bd5427f74c0dbf23c21240b2e17d from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:31.629171 7467 raft_consensus.cc:357] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.629776 7467 raft_consensus.cc:383] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.630017 7467 raft_consensus.cc:738] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.640452 7467 consensus_queue.cc:260] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.651580 7569 tablet_bootstrap.cc:492] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:31.653674 7467 ts_tablet_manager.cc:1428] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.026s user 0.002s sys 0.003s
I20250626 19:58:31.666257 7616 raft_consensus.cc:491] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:31.666824 7616 raft_consensus.cc:513] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.668676 7616 leader_election.cc:290] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:31.670269 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87b3b241727348a19c66c4ffcb0a675e" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:31.670706 7467 tablet_bootstrap.cc:492] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:31.670884 7215 raft_consensus.cc:2466] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:31.671850 7569 tablet_bootstrap.cc:654] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:31.674324 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87b3b241727348a19c66c4ffcb0a675e" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:31.675083 7073 raft_consensus.cc:2466] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:31.676936 7293 leader_election.cc:304] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:31.678483 7616 raft_consensus.cc:2802] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:31.678966 7616 raft_consensus.cc:491] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:31.679401 7616 raft_consensus.cc:3058] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:31.679926 7568 tablet_bootstrap.cc:492] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:31.680356 7568 ts_tablet_manager.cc:1397] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.103s user 0.008s sys 0.015s
I20250626 19:58:31.682818 7568 raft_consensus.cc:357] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.683413 7568 raft_consensus.cc:383] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.683645 7568 raft_consensus.cc:738] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.684267 7568 consensus_queue.cc:260] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.688743 7467 tablet_bootstrap.cc:654] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:31.691694 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 9216bd5427f74c0dbf23c21240b2e17d with delete_type TABLET_DATA_DELETED (replaced by tablet b65e085b67684ca18c95730b8a818286) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:31.692409 7559 tablet_replica.cc:331] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:31.693012 7559 raft_consensus.cc:2241] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:31.693394 7559 raft_consensus.cc:2270] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:31.694499 7568 ts_tablet_manager.cc:1428] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250626 19:58:31.695514 7559 ts_tablet_manager.cc:1905] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:31.701970 7568 tablet_bootstrap.cc:492] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:31.709126 7616 raft_consensus.cc:513] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.725211 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87b3b241727348a19c66c4ffcb0a675e" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:31.725840 7073 raft_consensus.cc:3058] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:31.727804 7560 ts_tablet_manager.cc:1918] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:31.728204 7560 log.cc:1199] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/131f263d07394c0d90edcc5637e3af51
I20250626 19:58:31.728729 7467 tablet_bootstrap.cc:492] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:31.728864 7568 tablet_bootstrap.cc:654] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:31.729132 7467 ts_tablet_manager.cc:1397] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.059s user 0.007s sys 0.003s
I20250626 19:58:31.731839 7560 ts_tablet_manager.cc:1939] T 131f263d07394c0d90edcc5637e3af51 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:31.731891 7467 raft_consensus.cc:357] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.732532 7467 raft_consensus.cc:383] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.732970 7073 raft_consensus.cc:2466] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:31.734112 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 131f263d07394c0d90edcc5637e3af51 (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:31.734184 7294 leader_election.cc:304] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:31.738781 7569 tablet_bootstrap.cc:492] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:31.739216 7569 ts_tablet_manager.cc:1397] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.088s user 0.006s sys 0.004s
I20250626 19:58:31.741333 7569 raft_consensus.cc:357] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.742869 7560 tablet_replica.cc:331] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:31.745015 7560 raft_consensus.cc:2241] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:31.752472 7569 raft_consensus.cc:383] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.752607 7616 leader_election.cc:290] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:31.752769 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87b3b241727348a19c66c4ffcb0a675e" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:31.753012 7569 raft_consensus.cc:738] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.753151 7616 raft_consensus.cc:2802] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:31.753330 7215 raft_consensus.cc:3058] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:31.753525 7616 raft_consensus.cc:695] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:31.753690 7560 raft_consensus.cc:2270] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:31.754310 7616 consensus_queue.cc:237] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.755399 7560 ts_tablet_manager.cc:1905] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:31.732821 7467 raft_consensus.cc:738] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.756256 7467 consensus_queue.cc:260] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.759449 7569 consensus_queue.cc:260] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.761291 7215 raft_consensus.cc:2466] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:31.762063 7568 tablet_bootstrap.cc:492] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:31.762435 7568 ts_tablet_manager.cc:1397] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.002s
I20250626 19:58:31.766825 7568 raft_consensus.cc:357] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.767422 7568 raft_consensus.cc:383] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.767673 7568 raft_consensus.cc:738] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.769517 7467 ts_tablet_manager.cc:1428] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.040s user 0.007s sys 0.001s
I20250626 19:58:31.779258 7573 raft_consensus.cc:491] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:31.779698 7573 raft_consensus.cc:513] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.781492 7573 leader_election.cc:290] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:31.782488 7559 ts_tablet_manager.cc:1918] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:31.782963 7559 log.cc:1199] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/9216bd5427f74c0dbf23c21240b2e17d
I20250626 19:58:31.783686 7559 ts_tablet_manager.cc:1939] T 9216bd5427f74c0dbf23c21240b2e17d P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:31.783959 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30bf36fa65414141a2e633ce72a9d7c4" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:31.784055 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30bf36fa65414141a2e633ce72a9d7c4" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:31.784615 7215 raft_consensus.cc:2466] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:31.784621 7360 raft_consensus.cc:2466] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:31.785849 7009 leader_election.cc:304] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:31.786999 7573 raft_consensus.cc:2802] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:31.787325 7573 raft_consensus.cc:491] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:31.787631 7573 raft_consensus.cc:3058] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:31.794827 7569 ts_tablet_manager.cc:1428] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.055s user 0.005s sys 0.001s
I20250626 19:58:31.796252 7573 raft_consensus.cc:513] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.799243 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30bf36fa65414141a2e633ce72a9d7c4" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:31.799804 7360 raft_consensus.cc:3058] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:31.800046 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30bf36fa65414141a2e633ce72a9d7c4" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:31.800560 7215 raft_consensus.cc:3058] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:31.805881 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 9216bd5427f74c0dbf23c21240b2e17d (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:31.806620 7360 raft_consensus.cc:2466] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:31.812997 7568 consensus_queue.cc:260] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.815387 7009 leader_election.cc:304] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:31.817376 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet ae04997b77924e8ba479c5c34ac822d7 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:31.819370 6709 catalog_manager.cc:5582] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } }
I20250626 19:58:31.820600 6709 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 87b3b241727348a19c66c4ffcb0a675e (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:31.826649 7215 raft_consensus.cc:2466] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:31.839232 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet 973b4134faa440469635b796847674ab from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:31.857646 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 973b4134faa440469635b796847674ab
I20250626 19:58:31.859273 6711 catalog_manager.cc:6652] ReplaceTablet: tablet 973b4134faa440469635b796847674ab deleted and replaced by tablet 6448edbc8a06486cabe00ca7f57e25ad
I20250626 19:58:31.860080 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 973b4134faa440469635b796847674ab with delete_type TABLET_DATA_DELETED (replaced by tablet 6448edbc8a06486cabe00ca7f57e25ad) from {username='slave'} at 127.0.0.1:54698
W20250626 19:58:31.861897 6703 catalog_manager.cc:4908] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 973b4134faa440469635b796847674ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 973b4134faa440469635b796847674ab already in progress: creating tablet
I20250626 19:58:31.863657 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 973b4134faa440469635b796847674ab with delete_type TABLET_DATA_DELETED (replaced by tablet 6448edbc8a06486cabe00ca7f57e25ad) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:31.864581 7473 tablet_replica.cc:331] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:31.865554 7473 raft_consensus.cc:2241] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:31.866139 7473 raft_consensus.cc:2270] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:31.868321 7473 ts_tablet_manager.cc:1905] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:31.873831 7569 tablet_bootstrap.cc:492] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:31.881470 7569 tablet_bootstrap.cc:654] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:31.883999 7568 ts_tablet_manager.cc:1428] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.121s user 0.007s sys 0.000s
I20250626 19:58:31.884881 7568 tablet_bootstrap.cc:492] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:31.891811 7573 leader_election.cc:290] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Requested vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:31.892484 7573 raft_consensus.cc:2802] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:31.892954 7573 raft_consensus.cc:695] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Becoming Leader. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Running, Role: LEADER
I20250626 19:58:31.893846 7573 consensus_queue.cc:237] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:31.903126 7009 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.131:33703, user_credentials={real_user=slave}} blocked reactor thread for 87843.2us
I20250626 19:58:31.917824 7473 ts_tablet_manager.cc:1918] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:31.918386 7473 log.cc:1199] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/973b4134faa440469635b796847674ab
I20250626 19:58:31.923790 7568 tablet_bootstrap.cc:654] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:31.939219 6704 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 180d9cb0d2874a8e9ad6d45fca6cb577 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 180d9cb0d2874a8e9ad6d45fca6cb577 consensus state has no leader
I20250626 19:58:31.947631 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet 973b4134faa440469635b796847674ab with delete_type TABLET_DATA_DELETED (replaced by tablet 6448edbc8a06486cabe00ca7f57e25ad) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:31.976774 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 6448edbc8a06486cabe00ca7f57e25ad (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:31.978026 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6448edbc8a06486cabe00ca7f57e25ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:31.978792 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 6448edbc8a06486cabe00ca7f57e25ad (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:31.979848 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6448edbc8a06486cabe00ca7f57e25ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:31.986017 7569 tablet_bootstrap.cc:492] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:31.986488 7569 ts_tablet_manager.cc:1397] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.113s user 0.012s sys 0.023s
I20250626 19:58:31.989576 7473 ts_tablet_manager.cc:1939] T 973b4134faa440469635b796847674ab P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:31.989389 7569 raft_consensus.cc:357] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.990698 7569 raft_consensus.cc:383] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:31.990985 7569 raft_consensus.cc:738] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:31.991525 7569 consensus_queue.cc:260] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:31.993299 7569 ts_tablet_manager.cc:1428] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250626 19:58:32.005383 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 973b4134faa440469635b796847674ab (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
I20250626 19:58:32.013168 6709 catalog_manager.cc:5582] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca reported cstate change: term changed from 0 to 1, leader changed from <none> to bbe151be01e24c82854d9c8c79c797ca (127.4.46.129). New cstate: current_term: 1 leader_uuid: "bbe151be01e24c82854d9c8c79c797ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:32.021919 7569 tablet_bootstrap.cc:492] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:32.026708 7193 tablet_service.cc:1468] Processing CreateTablet for tablet 6448edbc8a06486cabe00ca7f57e25ad (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:32.028455 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6448edbc8a06486cabe00ca7f57e25ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:32.030285 7568 tablet_bootstrap.cc:492] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:32.030845 7568 ts_tablet_manager.cc:1397] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.146s user 0.013s sys 0.010s
I20250626 19:58:32.038304 7568 raft_consensus.cc:357] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.040239 7568 raft_consensus.cc:383] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:32.045846 7568 raft_consensus.cc:738] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:32.046478 7568 consensus_queue.cc:260] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.059154 7467 tablet_bootstrap.cc:492] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:32.060014 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet 87b3b241727348a19c66c4ffcb0a675e from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:32.069586 7467 tablet_bootstrap.cc:654] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:32.080816 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 87b3b241727348a19c66c4ffcb0a675e
I20250626 19:58:32.084303 6711 catalog_manager.cc:6652] ReplaceTablet: tablet 87b3b241727348a19c66c4ffcb0a675e deleted and replaced by tablet e8cd3c1cc2de42e8a8503c66811f915f
I20250626 19:58:32.085074 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet ae04997b77924e8ba479c5c34ac822d7 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:32.085999 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 87b3b241727348a19c66c4ffcb0a675e with delete_type TABLET_DATA_DELETED (replaced by tablet e8cd3c1cc2de42e8a8503c66811f915f) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:32.086685 7559 tablet_replica.cc:331] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:32.086840 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 87b3b241727348a19c66c4ffcb0a675e with delete_type TABLET_DATA_DELETED (replaced by tablet e8cd3c1cc2de42e8a8503c66811f915f) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:32.087354 7559 raft_consensus.cc:2241] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:32.087844 7559 raft_consensus.cc:2270] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.089987 7559 ts_tablet_manager.cc:1905] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:58:32.094285 6704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.189:36417, user_credentials={real_user=slave}} blocked reactor thread for 44029.8us
I20250626 19:58:32.095347 7568 ts_tablet_manager.cc:1428] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.064s user 0.008s sys 0.000s
I20250626 19:58:32.096668 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 87b3b241727348a19c66c4ffcb0a675e with delete_type TABLET_DATA_DELETED (replaced by tablet e8cd3c1cc2de42e8a8503c66811f915f) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:32.097352 7473 tablet_replica.cc:331] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:32.098052 7473 raft_consensus.cc:2241] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:32.098692 7473 pending_rounds.cc:70] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7: Trying to abort 1 pending ops.
I20250626 19:58:32.098977 7473 pending_rounds.cc:77] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964566692499456 op_type: NO_OP noop_request { }
I20250626 19:58:32.099406 7473 raft_consensus.cc:2887] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:32.099843 7473 raft_consensus.cc:2270] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.100358 7569 tablet_bootstrap.cc:654] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:32.101351 7560 ts_tablet_manager.cc:1918] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:32.102589 7473 ts_tablet_manager.cc:1905] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:32.106981 7560 log.cc:1199] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/9216bd5427f74c0dbf23c21240b2e17d
I20250626 19:58:32.108073 7560 ts_tablet_manager.cc:1939] T 9216bd5427f74c0dbf23c21240b2e17d P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:32.112541 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 9216bd5427f74c0dbf23c21240b2e17d (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:32.113664 7560 tablet_replica.cc:331] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:32.114118 7467 tablet_bootstrap.cc:492] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:32.114447 7560 raft_consensus.cc:2241] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:32.114676 7467 ts_tablet_manager.cc:1397] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.005s
I20250626 19:58:32.116655 7560 raft_consensus.cc:2270] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.116981 7467 raft_consensus.cc:357] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.117571 7467 raft_consensus.cc:383] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:32.117820 7467 raft_consensus.cc:738] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:32.118680 7467 consensus_queue.cc:260] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.121740 7560 ts_tablet_manager.cc:1905] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:32.127851 7467 ts_tablet_manager.cc:1428] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250626 19:58:32.132778 7568 tablet_bootstrap.cc:492] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:32.138717 7568 tablet_bootstrap.cc:654] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:32.139642 7569 tablet_bootstrap.cc:492] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:32.140136 7569 ts_tablet_manager.cc:1397] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.118s user 0.007s sys 0.009s
I20250626 19:58:32.142477 7569 raft_consensus.cc:357] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.143307 7569 raft_consensus.cc:383] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:32.143626 7569 raft_consensus.cc:738] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:32.144315 7569 consensus_queue.cc:260] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.153410 7569 ts_tablet_manager.cc:1428] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.013s user 0.002s sys 0.003s
I20250626 19:58:32.154407 7569 tablet_bootstrap.cc:492] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:32.156318 7568 tablet_bootstrap.cc:492] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:32.156852 7568 ts_tablet_manager.cc:1397] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.001s
I20250626 19:58:32.159943 7569 tablet_bootstrap.cc:654] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:32.163113 7568 raft_consensus.cc:357] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.163938 7568 raft_consensus.cc:383] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:32.164247 7568 raft_consensus.cc:738] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:32.164964 7568 consensus_queue.cc:260] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.169013 7568 ts_tablet_manager.cc:1428] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20250626 19:58:32.180840 7560 ts_tablet_manager.cc:1918] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:32.187369 7560 log.cc:1199] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/1d7a74392e9d475588838a811011e9d6
I20250626 19:58:32.188271 7560 ts_tablet_manager.cc:1939] T 1d7a74392e9d475588838a811011e9d6 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:32.190945 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 1d7a74392e9d475588838a811011e9d6 (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
I20250626 19:58:32.197618 7560 tablet_replica.cc:331] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:32.199388 7560 raft_consensus.cc:2241] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:32.199939 7569 tablet_bootstrap.cc:492] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:32.200521 7569 ts_tablet_manager.cc:1397] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.046s user 0.030s sys 0.007s
I20250626 19:58:32.207083 7560 raft_consensus.cc:2270] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.209745 7569 raft_consensus.cc:357] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.212324 7560 ts_tablet_manager.cc:1905] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:58:32.215581 6711 catalog_manager.cc:5415] Ignoring report from unknown tablet 973b4134faa440469635b796847674ab
I20250626 19:58:32.220891 7569 raft_consensus.cc:383] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:32.221297 7569 raft_consensus.cc:738] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:32.214985 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 973b4134faa440469635b796847674ab with delete_type TABLET_DATA_DELETED (replaced by tablet 6448edbc8a06486cabe00ca7f57e25ad) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:32.222051 7569 consensus_queue.cc:260] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.223965 7569 ts_tablet_manager.cc:1428] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.016s user 0.004s sys 0.000s
W20250626 19:58:32.224043 6711 catalog_manager.cc:5415] Ignoring report from unknown tablet 973b4134faa440469635b796847674ab
I20250626 19:58:32.228611 7560 ts_tablet_manager.cc:1918] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:32.229017 7560 log.cc:1199] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/973b4134faa440469635b796847674ab
I20250626 19:58:32.229684 7560 ts_tablet_manager.cc:1939] T 973b4134faa440469635b796847674ab P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:32.231813 7640 raft_consensus.cc:491] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:32.232059 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 973b4134faa440469635b796847674ab (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
I20250626 19:58:32.232237 7640 raft_consensus.cc:513] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.232757 7560 tablet_replica.cc:331] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:32.233489 7560 raft_consensus.cc:2241] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:32.234102 7640 leader_election.cc:290] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:32.235241 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6448edbc8a06486cabe00ca7f57e25ad" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:32.236075 7215 raft_consensus.cc:2466] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:32.236397 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6448edbc8a06486cabe00ca7f57e25ad" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:32.237053 7360 raft_consensus.cc:2466] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:32.237442 7006 leader_election.cc:304] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:32.238274 7640 raft_consensus.cc:2802] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:32.238718 7640 raft_consensus.cc:491] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:32.239127 7640 raft_consensus.cc:3058] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:32.244401 7559 ts_tablet_manager.cc:1918] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:32.246553 7640 raft_consensus.cc:513] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.247499 7560 raft_consensus.cc:2270] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.249243 7560 ts_tablet_manager.cc:1905] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:32.254976 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6448edbc8a06486cabe00ca7f57e25ad" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:32.255664 7215 raft_consensus.cc:3058] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:32.257169 7193 tablet_service.cc:1468] Processing CreateTablet for tablet e8cd3c1cc2de42e8a8503c66811f915f (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:32.258241 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8cd3c1cc2de42e8a8503c66811f915f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:32.258512 7473 ts_tablet_manager.cc:1918] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:32.258910 7473 log.cc:1199] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/87b3b241727348a19c66c4ffcb0a675e
I20250626 19:58:32.259559 7473 ts_tablet_manager.cc:1939] T 87b3b241727348a19c66c4ffcb0a675e P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:32.259438 7339 tablet_service.cc:1468] Processing CreateTablet for tablet e8cd3c1cc2de42e8a8503c66811f915f (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:32.259570 7051 tablet_service.cc:1468] Processing CreateTablet for tablet e8cd3c1cc2de42e8a8503c66811f915f (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:32.260501 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8cd3c1cc2de42e8a8503c66811f915f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:32.260612 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8cd3c1cc2de42e8a8503c66811f915f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:32.261178 7559 log.cc:1199] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/87b3b241727348a19c66c4ffcb0a675e
I20250626 19:58:32.261839 7559 ts_tablet_manager.cc:1939] T 87b3b241727348a19c66c4ffcb0a675e P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:32.270491 7215 raft_consensus.cc:2466] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:32.274271 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 87b3b241727348a19c66c4ffcb0a675e (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:32.275645 7006 leader_election.cc:304] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:32.276722 7573 raft_consensus.cc:2802] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:32.277139 7559 tablet_replica.cc:331] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:32.284443 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6448edbc8a06486cabe00ca7f57e25ad" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:32.285055 7360 raft_consensus.cc:3058] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:32.287370 7559 raft_consensus.cc:2241] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:32.287863 7559 raft_consensus.cc:2270] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.291544 7609 raft_consensus.cc:491] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:32.292097 7609 raft_consensus.cc:513] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.292906 7559 ts_tablet_manager.cc:1905] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:32.296298 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b65e085b67684ca18c95730b8a818286" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:32.296916 7073 raft_consensus.cc:2466] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:32.298342 7150 leader_election.cc:304] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:32.299557 7609 leader_election.cc:290] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:32.301265 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b65e085b67684ca18c95730b8a818286" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:32.301877 7361 raft_consensus.cc:2466] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:32.303299 7609 raft_consensus.cc:2802] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:32.304168 7609 raft_consensus.cc:491] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:32.304632 7609 raft_consensus.cc:3058] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:32.305192 6942 hms_client.cc:305] Time spent drop HMS table: real 2.898s user 0.000s sys 0.000s
I20250626 19:58:32.306569 6718 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]
I20250626 19:58:32.316596 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 87b3b241727348a19c66c4ffcb0a675e (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:32.319242 7360 raft_consensus.cc:2466] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:32.324960 7640 leader_election.cc:290] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Requested vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:32.329313 7569 tablet_bootstrap.cc:492] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:32.334455 7569 tablet_bootstrap.cc:654] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:32.341967 7150 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:33129, user_credentials={real_user=slave}} blocked reactor thread for 43842.9us
I20250626 19:58:32.343319 7573 raft_consensus.cc:695] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Becoming Leader. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Running, Role: LEADER
I20250626 19:58:32.344251 7573 consensus_queue.cc:237] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:32.345095 6705 catalog_manager.cc:5001] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): alter failed for tablet 87b3b241727348a19c66c4ffcb0a675e (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]),no further retry: Not found: Tablet not found: 87b3b241727348a19c66c4ffcb0a675e
I20250626 19:58:32.346047 7559 ts_tablet_manager.cc:1918] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:32.346400 7559 log.cc:1199] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/973b4134faa440469635b796847674ab
I20250626 19:58:32.343688 7653 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:32.347311 7653 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.352610 7609 raft_consensus.cc:513] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.354615 7559 ts_tablet_manager.cc:1939] T 973b4134faa440469635b796847674ab P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
W20250626 19:58:32.355877 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 19:58:32.357193 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 973b4134faa440469635b796847674ab (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
W20250626 19:58:32.359181 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 19:58:32.359577 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:32.360685 7569 tablet_bootstrap.cc:492] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:32.361079 7569 ts_tablet_manager.cc:1397] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.002s
I20250626 19:58:32.361639 7568 tablet_bootstrap.cc:492] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:32.361706 7560 ts_tablet_manager.cc:1918] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:32.362095 7560 log.cc:1199] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/87b3b241727348a19c66c4ffcb0a675e
I20250626 19:58:32.360678 7654 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:32.363255 7569 raft_consensus.cc:357] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.367190 7568 tablet_bootstrap.cc:654] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:32.365988 7569 raft_consensus.cc:383] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:32.367921 7569 raft_consensus.cc:738] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:32.368472 7569 consensus_queue.cc:260] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.369064 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b65e085b67684ca18c95730b8a818286" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:32.369812 7073 raft_consensus.cc:3058] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:32.370380 7569 ts_tablet_manager.cc:1428] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250626 19:58:32.382076 7568 tablet_bootstrap.cc:492] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:32.382452 7568 ts_tablet_manager.cc:1397] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20250626 19:58:32.384658 7568 raft_consensus.cc:357] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.385242 7568 raft_consensus.cc:383] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:32.385505 7568 raft_consensus.cc:738] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:32.388481 6709 catalog_manager.cc:5582] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca reported cstate change: term changed from 0 to 1, leader changed from <none> to bbe151be01e24c82854d9c8c79c797ca (127.4.46.129). New cstate: current_term: 1 leader_uuid: "bbe151be01e24c82854d9c8c79c797ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:32.391299 7568 consensus_queue.cc:260] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.393304 7568 ts_tablet_manager.cc:1428] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.011s user 0.001s sys 0.004s
I20250626 19:58:32.362704 7560 ts_tablet_manager.cc:1939] T 87b3b241727348a19c66c4ffcb0a675e P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:32.406311 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 87b3b241727348a19c66c4ffcb0a675e (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:32.407316 6711 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet e8cd3c1cc2de42e8a8503c66811f915f (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
W20250626 19:58:32.408768 6711 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet e8cd3c1cc2de42e8a8503c66811f915f (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e8cd3c1cc2de42e8a8503c66811f915f consensus state has no leader
I20250626 19:58:32.419277 7073 raft_consensus.cc:2466] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:32.425487 6711 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet e8cd3c1cc2de42e8a8503c66811f915f (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:32.430490 7653 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:32.433312 7609 leader_election.cc:290] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:32.435815 7467 tablet_bootstrap.cc:492] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:32.433218 7150 leader_election.cc:304] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:32.444276 6714 catalog_manager.cc:2149] created HMS catalog entry for table [id=8768a4a6b5824e7db4f71bd958c17f49]
I20250626 19:58:32.454905 7467 tablet_bootstrap.cc:654] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:32.461470 6714 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 8) took 5961 ms. Trace:
I20250626 19:58:32.461855 6714 rpcz_store.cc:276] 0626 19:58:26.499668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:26.500880 (+ 1212us) service_pool.cc:225] Handling call
0626 19:58:28.461150 (+1960270us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:28.463094 (+ 1944us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:32.444134 (+3981040us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:32.460589 (+ 16455us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:32.460937 (+ 348us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:32.461379 (+ 442us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:32.446358 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:32.446736 (+ 378us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:32.446791 (+ 55us) write_op.cc:435] Acquired schema lock
0626 19:58:32.446804 (+ 13us) tablet.cc:660] Decoding operations
0626 19:58:32.447083 (+ 279us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:32.447118 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:32.447139 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:32.447322 (+ 183us) tablet.cc:699] Row locks acquired
0626 19:58:32.447333 (+ 11us) write_op.cc:260] PREPARE: finished
0626 19:58:32.447441 (+ 108us) write_op.cc:270] Start()
0626 19:58:32.447518 (+ 77us) write_op.cc:276] Timestamp: P: 1750967912447257 usec, L: 0
0626 19:58:32.447534 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:58:32.448004 (+ 470us) log.cc:844] Serialized 432 byte log entry
0626 19:58:32.458507 (+ 10503us) op_driver.cc:464] REPLICATION: finished
0626 19:58:32.458985 (+ 478us) write_op.cc:301] APPLY: starting
0626 19:58:32.459041 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:32.459141 (+ 100us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:32.459228 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:32.459599 (+ 371us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:32.459731 (+ 132us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:32.459773 (+ 42us) write_op.cc:312] APPLY: finished
0626 19:58:32.459941 (+ 168us) log.cc:844] Serialized 28 byte log entry
0626 19:58:32.460081 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:32.460150 (+ 69us) write_op.cc:454] Released schema lock
0626 19:58:32.460181 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0626 19:58:32.460407 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:32.460433 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":3841405,"Hive Metastore.run_cpu_time_us":684,"Hive Metastore.run_wall_time_us":137550,"child_traces":[["op",{"apply.queue_time_us":321,"apply.run_cpu_time_us":1385,"apply.run_wall_time_us":1383,"num_ops":2,"prepare.queue_time_us":276,"prepare.run_cpu_time_us":1925,"prepare.run_wall_time_us":3706,"raft.queue_time_us":870,"raft.run_cpu_time_us":621,"raft.run_wall_time_us":621,"replication_time_us":10871}]]}
I20250626 19:58:32.468194 6714 master_service.cc:948] ReplaceTablet: received request to replace tablet 6448edbc8a06486cabe00ca7f57e25ad from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:32.478143 7467 tablet_bootstrap.cc:492] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:32.478631 7467 ts_tablet_manager.cc:1397] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.007s
I20250626 19:58:32.444592 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet b65e085b67684ca18c95730b8a818286 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:32.445683 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b65e085b67684ca18c95730b8a818286" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:32.480922 7360 raft_consensus.cc:3058] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:32.481446 7467 raft_consensus.cc:357] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.482264 7467 raft_consensus.cc:383] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:32.482568 7467 raft_consensus.cc:738] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:32.490890 7467 consensus_queue.cc:260] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.496309 7609 raft_consensus.cc:2802] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:32.496798 7609 raft_consensus.cc:695] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:32.501365 7609 consensus_queue.cc:237] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.514657 7467 ts_tablet_manager.cc:1428] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.036s user 0.005s sys 0.001s
I20250626 19:58:32.517514 6714 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 6448edbc8a06486cabe00ca7f57e25ad
I20250626 19:58:32.519017 6714 catalog_manager.cc:6652] ReplaceTablet: tablet 6448edbc8a06486cabe00ca7f57e25ad deleted and replaced by tablet caa506927e37438c864d01250977da36
I20250626 19:58:32.521499 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 6448edbc8a06486cabe00ca7f57e25ad with delete_type TABLET_DATA_DELETED (replaced by tablet caa506927e37438c864d01250977da36) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:32.522708 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet 6448edbc8a06486cabe00ca7f57e25ad with delete_type TABLET_DATA_DELETED (replaced by tablet caa506927e37438c864d01250977da36) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:32.523030 7559 tablet_replica.cc:331] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:32.523444 7560 tablet_replica.cc:331] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:32.523716 7559 raft_consensus.cc:2241] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:32.524349 7559 pending_rounds.cc:70] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca: Trying to abort 1 pending ops.
I20250626 19:58:32.524274 7560 raft_consensus.cc:2241] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:32.524595 7559 pending_rounds.cc:77] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964568971350016 op_type: NO_OP noop_request { }
I20250626 19:58:32.524789 7560 raft_consensus.cc:2270] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.524915 7559 raft_consensus.cc:2887] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:32.525075 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 6448edbc8a06486cabe00ca7f57e25ad with delete_type TABLET_DATA_DELETED (replaced by tablet caa506927e37438c864d01250977da36) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:32.525216 7559 raft_consensus.cc:2270] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.525717 7473 tablet_replica.cc:331] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:32.526328 7473 raft_consensus.cc:2241] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:32.526770 7473 raft_consensus.cc:2270] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.526839 7560 ts_tablet_manager.cc:1905] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:32.528077 7559 ts_tablet_manager.cc:1905] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:32.528630 7473 ts_tablet_manager.cc:1905] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:32.530768 7616 raft_consensus.cc:491] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:32.531178 7616 raft_consensus.cc:513] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.532981 7616 leader_election.cc:290] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:32.536924 7193 tablet_service.cc:1468] Processing CreateTablet for tablet f4c8f2d3d62248b28c753403e2a1b4f4 (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:32.538024 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4c8f2d3d62248b28c753403e2a1b4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:32.538868 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8cd3c1cc2de42e8a8503c66811f915f" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:32.539639 7073 raft_consensus.cc:2466] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:32.541119 7051 tablet_service.cc:1468] Processing CreateTablet for tablet f4c8f2d3d62248b28c753403e2a1b4f4 (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:32.542176 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4c8f2d3d62248b28c753403e2a1b4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:32.542084 7294 leader_election.cc:304] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:32.544991 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8cd3c1cc2de42e8a8503c66811f915f" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:32.545807 7215 raft_consensus.cc:2466] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:32.547888 7339 tablet_service.cc:1468] Processing CreateTablet for tablet f4c8f2d3d62248b28c753403e2a1b4f4 (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:32.549147 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4c8f2d3d62248b28c753403e2a1b4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:32.551812 7616 raft_consensus.cc:2802] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:32.552139 7616 raft_consensus.cc:491] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:32.552532 7616 raft_consensus.cc:3058] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:32.555994 7560 ts_tablet_manager.cc:1918] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:32.556423 7560 log.cc:1199] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/6448edbc8a06486cabe00ca7f57e25ad
I20250626 19:58:32.557190 7560 ts_tablet_manager.cc:1939] T 6448edbc8a06486cabe00ca7f57e25ad P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:32.580832 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 6448edbc8a06486cabe00ca7f57e25ad (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
I20250626 19:58:32.587357 6709 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet e8cd3c1cc2de42e8a8503c66811f915f (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:32.591005 6714 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet e8cd3c1cc2de42e8a8503c66811f915f (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:32.605983 7360 raft_consensus.cc:2466] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:32.609401 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet b65e085b67684ca18c95730b8a818286
I20250626 19:58:32.611304 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet b65e085b67684ca18c95730b8a818286 with delete_type TABLET_DATA_DELETED (replaced by tablet 3303f89f34464ad5a39b91293bf11455) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:32.613309 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet b65e085b67684ca18c95730b8a818286 with delete_type TABLET_DATA_DELETED (replaced by tablet 3303f89f34464ad5a39b91293bf11455) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:32.614255 7560 tablet_replica.cc:331] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:32.615099 6711 catalog_manager.cc:6652] ReplaceTablet: tablet b65e085b67684ca18c95730b8a818286 deleted and replaced by tablet 3303f89f34464ad5a39b91293bf11455
I20250626 19:58:32.614568 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet b65e085b67684ca18c95730b8a818286 with delete_type TABLET_DATA_DELETED (replaced by tablet 3303f89f34464ad5a39b91293bf11455) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:32.618860 7616 raft_consensus.cc:513] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.635147 7569 tablet_bootstrap.cc:492] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:32.642591 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8cd3c1cc2de42e8a8503c66811f915f" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:32.643208 7073 raft_consensus.cc:3058] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:32.651948 7569 tablet_bootstrap.cc:654] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:32.650729 7073 raft_consensus.cc:2466] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:32.653414 7294 leader_election.cc:304] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:32.658147 7473 ts_tablet_manager.cc:1918] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:32.659488 7662 raft_consensus.cc:2802] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:32.661676 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8cd3c1cc2de42e8a8503c66811f915f" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:32.662235 7215 raft_consensus.cc:3058] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:32.662998 7559 ts_tablet_manager.cc:1918] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:32.663345 7559 log.cc:1199] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/6448edbc8a06486cabe00ca7f57e25ad
I20250626 19:58:32.664278 7559 ts_tablet_manager.cc:1939] T 6448edbc8a06486cabe00ca7f57e25ad P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:32.672369 7467 tablet_bootstrap.cc:492] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:32.673085 7662 raft_consensus.cc:695] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:32.673924 7662 consensus_queue.cc:237] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.674923 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 6448edbc8a06486cabe00ca7f57e25ad (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
I20250626 19:58:32.675386 6709 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_6a693c4bae214626a4050c7a94276f98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:32.677294 7215 raft_consensus.cc:2466] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:32.678741 7559 tablet_replica.cc:331] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:32.679383 7559 raft_consensus.cc:2241] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:32.679826 7559 raft_consensus.cc:2270] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.680153 7616 leader_election.cc:290] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:32.681517 7467 tablet_bootstrap.cc:654] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:32.685557 7568 tablet_bootstrap.cc:492] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:32.690822 7560 raft_consensus.cc:2241] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:32.691193 7568 tablet_bootstrap.cc:654] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:32.695261 7560 pending_rounds.cc:70] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a: Trying to abort 1 pending ops.
I20250626 19:58:32.695526 7560 pending_rounds.cc:77] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964570103603200 op_type: NO_OP noop_request { }
I20250626 19:58:32.695894 7560 raft_consensus.cc:2887] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:32.696216 7560 raft_consensus.cc:2270] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.696897 7559 ts_tablet_manager.cc:1905] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:32.706879 7560 ts_tablet_manager.cc:1905] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:32.713285 6711 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet e8cd3c1cc2de42e8a8503c66811f915f (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:32.715998 7051 tablet_service.cc:1468] Processing CreateTablet for tablet caa506927e37438c864d01250977da36 (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:32.716305 7339 tablet_service.cc:1468] Processing CreateTablet for tablet caa506927e37438c864d01250977da36 (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:32.717113 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa506927e37438c864d01250977da36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:32.717317 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa506927e37438c864d01250977da36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:32.723275 7193 tablet_service.cc:1468] Processing CreateTablet for tablet caa506927e37438c864d01250977da36 (DEFAULT_TABLE table=default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:32.724327 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa506927e37438c864d01250977da36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:32.730226 6714 master_service.cc:948] ReplaceTablet: received request to replace tablet f4c8f2d3d62248b28c753403e2a1b4f4 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:32.732676 7569 tablet_bootstrap.cc:492] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:32.733158 7569 ts_tablet_manager.cc:1397] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.098s user 0.011s sys 0.001s
I20250626 19:58:32.735620 7569 raft_consensus.cc:357] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:32.736191 7569 raft_consensus.cc:383] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:32.736435 7569 raft_consensus.cc:738] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:32.736980 7569 consensus_queue.cc:260] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:32.739843 7560 ts_tablet_manager.cc:1918] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:32.740204 7560 log.cc:1199] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/b65e085b67684ca18c95730b8a818286
I20250626 19:58:32.740876 7560 ts_tablet_manager.cc:1939] T b65e085b67684ca18c95730b8a818286 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:32.741987 7467 tablet_bootstrap.cc:492] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:32.742515 7467 ts_tablet_manager.cc:1397] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.070s user 0.014s sys 0.010s
I20250626 19:58:32.746007 7568 tablet_bootstrap.cc:492] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:32.745759 7467 raft_consensus.cc:357] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:32.746439 7568 ts_tablet_manager.cc:1397] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.001s
I20250626 19:58:32.746521 7467 raft_consensus.cc:383] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:32.748626 7568 raft_consensus.cc:357] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:32.749241 7568 raft_consensus.cc:383] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:32.749487 7568 raft_consensus.cc:738] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:32.749760 7467 raft_consensus.cc:738] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:32.750536 7467 consensus_queue.cc:260] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:32.753211 7568 consensus_queue.cc:260] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:32.755146 7568 ts_tablet_manager.cc:1428] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250626 19:58:32.658615 7473 log.cc:1199] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/6448edbc8a06486cabe00ca7f57e25ad
I20250626 19:58:32.765380 7473 ts_tablet_manager.cc:1939] T 6448edbc8a06486cabe00ca7f57e25ad P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:32.767836 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 6448edbc8a06486cabe00ca7f57e25ad (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
I20250626 19:58:32.768472 7467 ts_tablet_manager.cc:1428] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.026s user 0.004s sys 0.002s
I20250626 19:58:32.768692 7473 tablet_replica.cc:331] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:32.769973 7473 raft_consensus.cc:2241] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:32.770463 7473 raft_consensus.cc:2270] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.772847 6714 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet f4c8f2d3d62248b28c753403e2a1b4f4
I20250626 19:58:32.774808 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet f4c8f2d3d62248b28c753403e2a1b4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet db3cbdf7316149f8bf224bafd67461a6) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:32.775496 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet f4c8f2d3d62248b28c753403e2a1b4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet db3cbdf7316149f8bf224bafd67461a6) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:32.775696 6714 catalog_manager.cc:6652] ReplaceTablet: tablet f4c8f2d3d62248b28c753403e2a1b4f4 deleted and replaced by tablet db3cbdf7316149f8bf224bafd67461a6
I20250626 19:58:32.776232 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet f4c8f2d3d62248b28c753403e2a1b4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet db3cbdf7316149f8bf224bafd67461a6) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:32.793955 7569 ts_tablet_manager.cc:1428] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.060s user 0.003s sys 0.004s
I20250626 19:58:32.804821 6713 catalog_manager.cc:2149] created HMS catalog entry for table [id=8c650357a157442e8b725ae57770ac8d]
I20250626 19:58:32.825172 7560 tablet_replica.cc:331] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:32.825490 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet b65e085b67684ca18c95730b8a818286 (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:32.825860 7560 raft_consensus.cc:2241] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:32.826270 7560 raft_consensus.cc:2270] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.828873 7569 tablet_bootstrap.cc:492] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:32.835897 7560 ts_tablet_manager.cc:1905] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:32.838373 6711 catalog_manager.cc:5582] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } }
I20250626 19:58:32.839509 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 3303f89f34464ad5a39b91293bf11455 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:32.839669 6711 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet e8cd3c1cc2de42e8a8503c66811f915f (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:32.840498 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3303f89f34464ad5a39b91293bf11455. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:32.842964 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 3303f89f34464ad5a39b91293bf11455 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:32.844298 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 3303f89f34464ad5a39b91293bf11455 (DEFAULT_TABLE table=default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:32.845311 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3303f89f34464ad5a39b91293bf11455. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:32.848889 7569 tablet_bootstrap.cc:654] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:32.839473 7559 ts_tablet_manager.cc:1918] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:32.852116 7559 log.cc:1199] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/b65e085b67684ca18c95730b8a818286
I20250626 19:58:32.852751 7559 ts_tablet_manager.cc:1939] T b65e085b67684ca18c95730b8a818286 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:32.862855 6713 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 10) took 6313 ms. Trace:
I20250626 19:58:32.860349 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet b65e085b67684ca18c95730b8a818286 (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:32.863195 6713 rpcz_store.cc:276] 0626 19:58:26.549930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:26.550034 (+ 104us) service_pool.cc:225] Handling call
0626 19:58:28.465985 (+1915951us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:28.468176 (+ 2191us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:32.804644 (+4336468us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:32.862065 (+ 57421us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:32.862389 (+ 324us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:32.862795 (+ 406us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:32.807792 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:32.808390 (+ 598us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:32.808413 (+ 23us) write_op.cc:435] Acquired schema lock
0626 19:58:32.808425 (+ 12us) tablet.cc:660] Decoding operations
0626 19:58:32.808808 (+ 383us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:32.808842 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:32.808887 (+ 45us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:32.809190 (+ 303us) tablet.cc:699] Row locks acquired
0626 19:58:32.809226 (+ 36us) write_op.cc:260] PREPARE: finished
0626 19:58:32.809355 (+ 129us) write_op.cc:270] Start()
0626 19:58:32.809488 (+ 133us) write_op.cc:276] Timestamp: P: 1750967912809255 usec, L: 0
0626 19:58:32.809507 (+ 19us) op_driver.cc:348] REPLICATION: starting
0626 19:58:32.810202 (+ 695us) log.cc:844] Serialized 432 byte log entry
0626 19:58:32.857263 (+ 47061us) op_driver.cc:464] REPLICATION: finished
0626 19:58:32.857686 (+ 423us) write_op.cc:301] APPLY: starting
0626 19:58:32.857752 (+ 66us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:32.857877 (+ 125us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:32.857965 (+ 88us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:32.861145 (+ 3180us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:32.861303 (+ 158us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:32.861361 (+ 58us) write_op.cc:312] APPLY: finished
0626 19:58:32.861589 (+ 228us) log.cc:844] Serialized 28 byte log entry
0626 19:58:32.861745 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:32.861819 (+ 74us) write_op.cc:454] Released schema lock
0626 19:58:32.861855 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0626 19:58:32.862107 (+ 252us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:32.862146 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":3974733,"Hive Metastore.run_cpu_time_us":664,"Hive Metastore.run_wall_time_us":360443,"child_traces":[["op",{"apply.queue_time_us":269,"apply.run_cpu_time_us":1746,"apply.run_wall_time_us":4382,"num_ops":2,"prepare.queue_time_us":1257,"prepare.run_cpu_time_us":6860,"prepare.run_wall_time_us":24252,"raft.queue_time_us":120,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":13,"replication_time_us":47668}]]}
I20250626 19:58:32.870244 7568 tablet_bootstrap.cc:492] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:32.870968 7559 tablet_replica.cc:331] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:32.871627 7559 raft_consensus.cc:2241] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:32.872049 7559 raft_consensus.cc:2270] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.874349 6714 master_service.cc:948] ReplaceTablet: received request to replace tablet b65e085b67684ca18c95730b8a818286 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:32.874295 7653 consensus_queue.cc:1035] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:32.876091 7568 tablet_bootstrap.cc:654] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:32.876328 7559 ts_tablet_manager.cc:1905] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:32.881824 7473 ts_tablet_manager.cc:1905] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:32.884171 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3303f89f34464ad5a39b91293bf11455. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:32.897179 7568 tablet_bootstrap.cc:492] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:32.897552 7568 ts_tablet_manager.cc:1397] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.002s
I20250626 19:58:32.902135 7568 raft_consensus.cc:357] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.902709 7568 raft_consensus.cc:383] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:32.903039 7568 raft_consensus.cc:738] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:32.907177 7568 consensus_queue.cc:260] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.908082 7559 ts_tablet_manager.cc:1918] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:32.908423 7559 log.cc:1199] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/f4c8f2d3d62248b28c753403e2a1b4f4
I20250626 19:58:32.913733 7559 ts_tablet_manager.cc:1939] T f4c8f2d3d62248b28c753403e2a1b4f4 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:32.915683 7568 ts_tablet_manager.cc:1428] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.015s user 0.004s sys 0.001s
I20250626 19:58:32.916482 7568 tablet_bootstrap.cc:492] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:32.924340 7467 tablet_bootstrap.cc:492] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:32.929176 7467 tablet_bootstrap.cc:654] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:32.935830 6715 catalog_manager.cc:2149] created HMS catalog entry for table [id=2fd44676faf246d285e450e5918c169a]
I20250626 19:58:32.936853 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet f4c8f2d3d62248b28c753403e2a1b4f4 (table default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:32.942783 7560 ts_tablet_manager.cc:1918] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:32.944931 7560 log.cc:1199] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/f4c8f2d3d62248b28c753403e2a1b4f4
I20250626 19:58:32.945915 7560 ts_tablet_manager.cc:1939] T f4c8f2d3d62248b28c753403e2a1b4f4 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:32.950071 7568 tablet_bootstrap.cc:654] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:32.952329 7467 tablet_bootstrap.cc:492] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:32.952747 7467 ts_tablet_manager.cc:1397] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.005s
I20250626 19:58:32.953271 7609 raft_consensus.cc:491] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:32.953819 7609 raft_consensus.cc:513] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.954937 7467 raft_consensus.cc:357] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.955570 7467 raft_consensus.cc:383] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:32.955528 7609 leader_election.cc:290] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:32.955806 7467 raft_consensus.cc:738] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:32.956419 7467 consensus_queue.cc:260] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.957229 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet f4c8f2d3d62248b28c753403e2a1b4f4 (table default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:32.958295 7467 ts_tablet_manager.cc:1428] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250626 19:58:32.965792 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "caa506927e37438c864d01250977da36" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:32.966414 7360 raft_consensus.cc:2466] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:32.967882 7473 ts_tablet_manager.cc:1918] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:32.968238 7473 log.cc:1199] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/b65e085b67684ca18c95730b8a818286
I20250626 19:58:32.968106 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "caa506927e37438c864d01250977da36" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:32.968806 7473 ts_tablet_manager.cc:1939] T b65e085b67684ca18c95730b8a818286 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
W20250626 19:58:32.969655 7150 leader_election.cc:343] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:32.970741 7152 leader_election.cc:304] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:32.972241 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet b65e085b67684ca18c95730b8a818286 (table default.table_d68d3f0aa49c4afb89ce351c9baf0a2d [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:32.973383 7609 raft_consensus.cc:2802] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:32.973831 7609 raft_consensus.cc:491] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:32.974262 7609 raft_consensus.cc:3058] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:32.984117 7568 tablet_bootstrap.cc:492] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:32.984506 7568 ts_tablet_manager.cc:1397] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.004s
I20250626 19:58:32.985498 7473 tablet_replica.cc:331] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:32.986181 7473 raft_consensus.cc:2241] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:32.987035 7568 raft_consensus.cc:357] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.987599 7568 raft_consensus.cc:383] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:32.987826 7568 raft_consensus.cc:738] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:32.988354 7568 consensus_queue.cc:260] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:32.990182 7568 ts_tablet_manager.cc:1428] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20250626 19:58:32.986601 7473 raft_consensus.cc:2270] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:32.992401 7473 ts_tablet_manager.cc:1905] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:33.005467 7473 ts_tablet_manager.cc:1918] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:33.005849 7473 log.cc:1199] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/f4c8f2d3d62248b28c753403e2a1b4f4
I20250626 19:58:33.006414 7473 ts_tablet_manager.cc:1939] T f4c8f2d3d62248b28c753403e2a1b4f4 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:33.008495 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet f4c8f2d3d62248b28c753403e2a1b4f4 (table default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:33.036015 6713 master_service.cc:948] ReplaceTablet: received request to replace tablet ae04997b77924e8ba479c5c34ac822d7 from {username='slave'} at 127.0.0.1:42822
W20250626 19:58:33.039664 6704 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e8cd3c1cc2de42e8a8503c66811f915f (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e8cd3c1cc2de42e8a8503c66811f915f consensus state has no leader
I20250626 19:58:33.058295 6715 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 9) took 6508 ms. Trace:
I20250626 19:58:33.058640 6715 rpcz_store.cc:276] 0626 19:58:26.549568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:26.554002 (+ 4434us) service_pool.cc:225] Handling call
0626 19:58:28.470190 (+1916188us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:28.472032 (+ 1842us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:32.935682 (+4463650us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:33.057575 (+121893us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:33.057867 (+ 292us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:33.058237 (+ 370us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:33.032884 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:33.033323 (+ 439us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:33.033347 (+ 24us) write_op.cc:435] Acquired schema lock
0626 19:58:33.033379 (+ 32us) tablet.cc:660] Decoding operations
0626 19:58:33.033644 (+ 265us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:33.033678 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:33.033699 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:33.048373 (+ 14674us) tablet.cc:699] Row locks acquired
0626 19:58:33.048387 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:58:33.048483 (+ 96us) write_op.cc:270] Start()
0626 19:58:33.048556 (+ 73us) write_op.cc:276] Timestamp: P: 1750967913048478 usec, L: 0
0626 19:58:33.048572 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:58:33.049003 (+ 431us) log.cc:844] Serialized 432 byte log entry
0626 19:58:33.055693 (+ 6690us) op_driver.cc:464] REPLICATION: finished
0626 19:58:33.056073 (+ 380us) write_op.cc:301] APPLY: starting
0626 19:58:33.056123 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:33.056216 (+ 93us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:33.056312 (+ 96us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:33.056634 (+ 322us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:33.056753 (+ 119us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:33.056792 (+ 39us) write_op.cc:312] APPLY: finished
0626 19:58:33.056963 (+ 171us) log.cc:844] Serialized 28 byte log entry
0626 19:58:33.057104 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:33.057164 (+ 60us) write_op.cc:454] Released schema lock
0626 19:58:33.057193 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0626 19:58:33.058852 (+ 1659us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:33.058876 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":4331452,"Hive Metastore.run_cpu_time_us":584,"Hive Metastore.run_wall_time_us":130317,"child_traces":[["op",{"apply.queue_time_us":244,"apply.run_cpu_time_us":1316,"apply.run_wall_time_us":1316,"num_ops":2,"prepare.queue_time_us":95444,"prepare.run_cpu_time_us":2028,"prepare.run_wall_time_us":25958,"raft.queue_time_us":269,"raft.run_cpu_time_us":582,"raft.run_wall_time_us":583,"replication_time_us":7020}]]}
I20250626 19:58:33.082051 7609 raft_consensus.cc:513] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.084249 6715 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_8a5f3c205ce2406e8b57c6f09619cb05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:33.085211 7653 consensus_queue.cc:1035] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:33.088158 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "caa506927e37438c864d01250977da36" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
W20250626 19:58:33.089325 7150 leader_election.cc:343] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:33.093919 7609 leader_election.cc:290] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:33.094933 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "caa506927e37438c864d01250977da36" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:33.095443 7361 raft_consensus.cc:3058] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:33.102051 7361 raft_consensus.cc:2466] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:33.103225 7152 leader_election.cc:304] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:33.106833 7467 tablet_bootstrap.cc:492] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:33.111948 7467 tablet_bootstrap.cc:654] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:33.126368 7609 raft_consensus.cc:2802] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:33.130132 7569 tablet_bootstrap.cc:492] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:33.130524 7569 ts_tablet_manager.cc:1397] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.302s user 0.021s sys 0.050s
I20250626 19:58:33.131119 7467 tablet_bootstrap.cc:492] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:33.131496 7467 ts_tablet_manager.cc:1397] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.003s
I20250626 19:58:33.133064 7569 raft_consensus.cc:357] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.133836 7569 raft_consensus.cc:383] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:33.133692 7467 raft_consensus.cc:357] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.134286 7467 raft_consensus.cc:383] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:33.140224 7196 tablet_service.cc:1468] Processing CreateTablet for tablet db3cbdf7316149f8bf224bafd67461a6 (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:33.141220 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db3cbdf7316149f8bf224bafd67461a6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:58:33.141149 6944 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3303f89f34464ad5a39b91293bf11455 consensus state has no leader
I20250626 19:58:33.143087 7051 tablet_service.cc:1468] Processing CreateTablet for tablet db3cbdf7316149f8bf224bafd67461a6 (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:33.143998 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db3cbdf7316149f8bf224bafd67461a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:33.146332 7569 raft_consensus.cc:738] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:33.148381 7569 consensus_queue.cc:260] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.157639 7609 raft_consensus.cc:695] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:33.158965 7609 consensus_queue.cc:237] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.161469 6944 catalog_manager.cc:2747] Deleting table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c] in response to Hive Metastore notification log event 7
I20250626 19:58:33.167286 7569 ts_tablet_manager.cc:1428] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20250626 19:58:33.174793 7569 tablet_bootstrap.cc:492] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:33.192783 7569 tablet_bootstrap.cc:654] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:33.134613 7467 raft_consensus.cc:738] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:33.200632 7467 consensus_queue.cc:260] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.207172 7467 ts_tablet_manager.cc:1428] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.075s user 0.006s sys 0.001s
I20250626 19:58:33.184530 7338 tablet_service.cc:1468] Processing CreateTablet for tablet db3cbdf7316149f8bf224bafd67461a6 (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:33.213941 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db3cbdf7316149f8bf224bafd67461a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:33.218266 6944 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 30bf36fa65414141a2e633ce72a9d7c4
I20250626 19:58:33.220140 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 30bf36fa65414141a2e633ce72a9d7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:33 UTC) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:33.220943 7559 tablet_replica.cc:331] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:33.224301 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet 30bf36fa65414141a2e633ce72a9d7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:33 UTC) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:33.224951 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 30bf36fa65414141a2e633ce72a9d7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:33 UTC) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:33.225157 7560 tablet_replica.cc:331] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:33.225576 7473 tablet_replica.cc:331] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:33.225814 7560 raft_consensus.cc:2241] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:33.226194 7473 raft_consensus.cc:2241] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:33.226253 7560 raft_consensus.cc:2270] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:33.226629 7473 raft_consensus.cc:2270] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:33.228766 7560 ts_tablet_manager.cc:1905] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:33.230620 7559 raft_consensus.cc:2241] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:33.232021 7559 raft_consensus.cc:2270] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:33.233800 7559 ts_tablet_manager.cc:1905] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:33.236718 7569 tablet_bootstrap.cc:492] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:33.237083 7569 ts_tablet_manager.cc:1397] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.062s user 0.009s sys 0.005s
I20250626 19:58:33.245998 6713 master_service.cc:948] ReplaceTablet: received request to replace tablet 734d9d3331b44fc49e1f64b8caf1e4d8 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:33.246428 7473 ts_tablet_manager.cc:1905] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:33.248739 7559 ts_tablet_manager.cc:1918] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:33.249135 7559 log.cc:1199] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/30bf36fa65414141a2e633ce72a9d7c4
I20250626 19:58:33.249907 7559 ts_tablet_manager.cc:1939] T 30bf36fa65414141a2e633ce72a9d7c4 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:33.250116 7569 raft_consensus.cc:357] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.250720 7569 raft_consensus.cc:383] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:33.250978 7569 raft_consensus.cc:738] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:33.252450 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 30bf36fa65414141a2e633ce72a9d7c4 (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:33.258637 7560 ts_tablet_manager.cc:1918] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:33.259140 7560 log.cc:1199] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/30bf36fa65414141a2e633ce72a9d7c4
I20250626 19:58:33.259920 7560 ts_tablet_manager.cc:1939] T 30bf36fa65414141a2e633ce72a9d7c4 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:33.258831 6714 master_service.cc:948] ReplaceTablet: received request to replace tablet 734d9d3331b44fc49e1f64b8caf1e4d8 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:33.262655 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 30bf36fa65414141a2e633ce72a9d7c4 (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:33.264026 7569 consensus_queue.cc:260] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.272711 7569 ts_tablet_manager.cc:1428] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.035s user 0.003s sys 0.003s
I20250626 19:58:33.275956 7569 tablet_bootstrap.cc:492] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:33.281704 7569 tablet_bootstrap.cc:654] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:33.281555 6714 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_76cdf24ffacc4d129b66c2fd6c0b93d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:33.289398 6711 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 3303f89f34464ad5a39b91293bf11455 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
I20250626 19:58:33.293155 6713 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 3303f89f34464ad5a39b91293bf11455 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
I20250626 19:58:33.296622 6711 catalog_manager.cc:5582] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130). New cstate: current_term: 1 leader_uuid: "315d304854c541bf8b5e6b1eea712d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:33.303277 7051 tablet_service.cc:1468] Processing CreateTablet for tablet b121b5cca8144d9ca9c303709083a101 (DEFAULT_TABLE table=default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:33.304221 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b121b5cca8144d9ca9c303709083a101. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:33.305917 7193 tablet_service.cc:1468] Processing CreateTablet for tablet b121b5cca8144d9ca9c303709083a101 (DEFAULT_TABLE table=default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:33.306941 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b121b5cca8144d9ca9c303709083a101. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:33.311651 7568 tablet_bootstrap.cc:492] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:33.317958 7568 tablet_bootstrap.cc:654] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:33.321740 7339 tablet_service.cc:1468] Processing CreateTablet for tablet b121b5cca8144d9ca9c303709083a101 (DEFAULT_TABLE table=default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:33.322702 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b121b5cca8144d9ca9c303709083a101. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:33.333196 7473 ts_tablet_manager.cc:1918] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:33.333616 7473 log.cc:1199] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/30bf36fa65414141a2e633ce72a9d7c4
I20250626 19:58:33.334326 7473 ts_tablet_manager.cc:1939] T 30bf36fa65414141a2e633ce72a9d7c4 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:33.334951 7568 tablet_bootstrap.cc:492] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:33.335436 7568 ts_tablet_manager.cc:1397] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.009s
I20250626 19:58:33.338071 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 73e216d7176e445dbd2b7a54845636eb (DEFAULT_TABLE table=default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:33.338246 7568 raft_consensus.cc:357] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.339063 7568 raft_consensus.cc:383] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:33.339385 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e216d7176e445dbd2b7a54845636eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:33.339375 7568 raft_consensus.cc:738] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:33.352795 7568 consensus_queue.cc:260] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.355579 7568 ts_tablet_manager.cc:1428] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20250626 19:58:33.369127 7569 tablet_bootstrap.cc:492] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:33.369210 6713 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 3303f89f34464ad5a39b91293bf11455 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
I20250626 19:58:33.369535 7569 ts_tablet_manager.cc:1397] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.094s user 0.032s sys 0.044s
I20250626 19:58:33.381755 7569 raft_consensus.cc:357] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:33.387877 6710 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a050ca8b9a9043ee87ca530228790f54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:33.394074 6716 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42822 (request call id 6) took 7151 ms. Trace:
I20250626 19:58:33.394397 7569 raft_consensus.cc:383] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:33.394680 7569 raft_consensus.cc:738] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:33.395366 7569 consensus_queue.cc:260] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.397208 7569 ts_tablet_manager.cc:1428] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.027s user 0.004s sys 0.000s
I20250626 19:58:33.398003 7569 tablet_bootstrap.cc:492] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:33.394395 6716 rpcz_store.cc:276] 0626 19:58:26.242157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:26.242277 (+ 120us) service_pool.cc:225] Handling call
0626 19:58:26.250737 (+ 8460us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b54000203d4. stack: 00007efd765d2483 00007efd765d2380 00007efd7614e574 00000000004fa174 00000000004fa109 00007efd7d4d4a70 00007efd7d558b1a 00007efd7f205fac 00007efd7f3003b6 00007efd7d025ca4 00007efd7d025c32 00007efd7d025b61 00007efd7d025adc 00007efd7d024d92 00007efd7b318dcc 00007efd7b318206
0626 19:58:26.285176 (+ 34439us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:33.391016 (+7105840us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:33.393029 (+ 2013us) catalog_manager.cc:3683] Apply alter schema
0626 19:58:33.393888 (+ 859us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":185034,"Hive Metastore.run_cpu_time_us":2441,"Hive Metastore.run_wall_time_us":1915795,"spinlock_wait_cycles":6202112}
I20250626 19:58:33.413425 7609 raft_consensus.cc:491] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:33.413868 7609 raft_consensus.cc:513] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.415640 7609 leader_election.cc:290] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:33.417011 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db3cbdf7316149f8bf224bafd67461a6" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:33.417562 7073 raft_consensus.cc:2466] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:33.418517 7150 leader_election.cc:304] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:33.419399 7609 raft_consensus.cc:2802] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:33.419785 7609 raft_consensus.cc:491] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:33.420090 7609 raft_consensus.cc:3058] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:33.426268 7609 raft_consensus.cc:513] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.428627 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db3cbdf7316149f8bf224bafd67461a6" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:33.429131 7073 raft_consensus.cc:3058] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:33.431720 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 73e216d7176e445dbd2b7a54845636eb (DEFAULT_TABLE table=default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:33.432686 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e216d7176e445dbd2b7a54845636eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:33.435415 7568 tablet_bootstrap.cc:492] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:33.435679 7073 raft_consensus.cc:2466] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:33.436678 7150 leader_election.cc:304] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:33.437506 7655 raft_consensus.cc:2802] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:33.441220 7568 tablet_bootstrap.cc:654] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:33.357932 7467 tablet_bootstrap.cc:492] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:33.418514 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db3cbdf7316149f8bf224bafd67461a6" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
W20250626 19:58:33.445796 7152 leader_election.cc:343] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:33.430357 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 73e216d7176e445dbd2b7a54845636eb (DEFAULT_TABLE table=default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:33.446575 7655 raft_consensus.cc:695] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:33.447129 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e216d7176e445dbd2b7a54845636eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:33.447378 7655 consensus_queue.cc:237] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.449705 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 30bf36fa65414141a2e633ce72a9d7c4 (table default.table_d0b4adacbfa84fbc8979bd9fe8d5bca7 [id=8b447518f0d84428b97e7ffee1a90a2c]) successfully deleted
I20250626 19:58:33.454362 7569 tablet_bootstrap.cc:654] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:33.456176 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db3cbdf7316149f8bf224bafd67461a6" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:33.456811 7467 tablet_bootstrap.cc:654] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:33.457574 7152 leader_election.cc:343] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:33.458109 7609 leader_election.cc:290] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:33.469393 7568 tablet_bootstrap.cc:492] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:33.470888 7568 ts_tablet_manager.cc:1397] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.000s
I20250626 19:58:33.475683 7568 raft_consensus.cc:357] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.476300 7568 raft_consensus.cc:383] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:33.476523 7568 raft_consensus.cc:738] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:33.478536 6716 master_service.cc:948] ReplaceTablet: received request to replace tablet b121b5cca8144d9ca9c303709083a101 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:33.481369 7568 consensus_queue.cc:260] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.487576 7568 ts_tablet_manager.cc:1428] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250626 19:58:33.495980 7467 tablet_bootstrap.cc:492] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:33.496367 7467 ts_tablet_manager.cc:1397] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.139s user 0.009s sys 0.000s
I20250626 19:58:33.498536 7467 raft_consensus.cc:357] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.499333 7467 raft_consensus.cc:383] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:33.499634 7467 raft_consensus.cc:738] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:33.511036 7467 consensus_queue.cc:260] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.514214 7569 tablet_bootstrap.cc:492] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:33.514590 7569 ts_tablet_manager.cc:1397] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.117s user 0.013s sys 0.000s
I20250626 19:58:33.516536 7569 raft_consensus.cc:357] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.517127 7569 raft_consensus.cc:383] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:33.517361 7569 raft_consensus.cc:738] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:33.519766 7655 raft_consensus.cc:491] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:33.520359 7569 consensus_queue.cc:260] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.520339 7655 raft_consensus.cc:513] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.522320 7655 leader_election.cc:290] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:33.524725 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3303f89f34464ad5a39b91293bf11455" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:33.525486 7073 raft_consensus.cc:2466] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:33.527040 7150 leader_election.cc:304] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:33.530545 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3303f89f34464ad5a39b91293bf11455" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:33.531148 7360 raft_consensus.cc:2466] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:33.531301 7467 ts_tablet_manager.cc:1428] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20250626 19:58:33.532194 7655 raft_consensus.cc:2802] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:33.532527 7655 raft_consensus.cc:491] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:33.532814 7655 raft_consensus.cc:3058] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:33.533860 7568 tablet_bootstrap.cc:492] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:33.538017 6716 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet b121b5cca8144d9ca9c303709083a101
I20250626 19:58:33.539789 6716 catalog_manager.cc:6652] ReplaceTablet: tablet b121b5cca8144d9ca9c303709083a101 deleted and replaced by tablet 0d55b03615694e3baa0f4b716a830128
I20250626 19:58:33.540864 7467 tablet_bootstrap.cc:492] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:33.549437 7568 tablet_bootstrap.cc:654] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:33.554800 7467 tablet_bootstrap.cc:654] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:33.555346 7673 raft_consensus.cc:491] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:33.555809 7673 raft_consensus.cc:513] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.557603 7673 leader_election.cc:290] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:33.560081 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b121b5cca8144d9ca9c303709083a101" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:33.560624 7216 raft_consensus.cc:2466] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:33.561564 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b121b5cca8144d9ca9c303709083a101" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
W20250626 19:58:33.562798 7009 leader_election.cc:343] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:33.563756 7006 leader_election.cc:304] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7
I20250626 19:58:33.566956 7673 raft_consensus.cc:2802] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:33.567281 7673 raft_consensus.cc:491] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:33.567631 7673 raft_consensus.cc:3058] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:33.572525 7568 tablet_bootstrap.cc:492] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:33.572774 7073 raft_consensus.cc:1273] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Refusing update from remote peer 297d2c913667408bae82668a557e06c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:58:33.572937 7568 ts_tablet_manager.cc:1397] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.001s
I20250626 19:58:33.575964 7569 ts_tablet_manager.cc:1428] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.061s user 0.003s sys 0.001s
I20250626 19:58:33.576000 7216 raft_consensus.cc:1273] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Refusing update from remote peer 297d2c913667408bae82668a557e06c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:58:33.575903 7568 raft_consensus.cc:357] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.576526 7568 raft_consensus.cc:383] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:33.576767 7568 raft_consensus.cc:738] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:33.577324 7568 consensus_queue.cc:260] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.578097 7662 consensus_queue.cc:1035] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:33.579193 7568 ts_tablet_manager.cc:1428] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250626 19:58:33.579946 7467 tablet_bootstrap.cc:492] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:33.580353 7467 ts_tablet_manager.cc:1397] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.000s
I20250626 19:58:33.583598 7467 raft_consensus.cc:357] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.587110 7673 raft_consensus.cc:513] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.584203 7467 raft_consensus.cc:383] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:33.584370 7662 consensus_queue.cc:1035] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:33.589576 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b121b5cca8144d9ca9c303709083a101" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:33.589907 7467 raft_consensus.cc:738] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:33.590085 7216 raft_consensus.cc:3058] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:33.590549 7467 consensus_queue.cc:260] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.592541 7467 ts_tablet_manager.cc:1428] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20250626 19:58:33.595631 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet b121b5cca8144d9ca9c303709083a101 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d55b03615694e3baa0f4b716a830128) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:33.597131 7216 raft_consensus.cc:2466] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:33.598124 7006 leader_election.cc:304] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:33.601822 7467 tablet_bootstrap.cc:492] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:33.546444 7655 raft_consensus.cc:513] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.609365 7653 raft_consensus.cc:2802] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:33.609760 7653 raft_consensus.cc:695] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Becoming Leader. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Running, Role: LEADER
I20250626 19:58:33.610452 7653 consensus_queue.cc:237] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.615339 7560 tablet_replica.cc:331] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:33.616175 7560 raft_consensus.cc:2241] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:33.616591 7560 raft_consensus.cc:2270] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:33.612004 7467 tablet_bootstrap.cc:654] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:33.618678 7560 ts_tablet_manager.cc:1905] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:33.619861 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet b121b5cca8144d9ca9c303709083a101 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d55b03615694e3baa0f4b716a830128) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:33.621127 7473 tablet_replica.cc:331] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:33.621877 7569 tablet_bootstrap.cc:492] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:33.628252 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3303f89f34464ad5a39b91293bf11455" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:33.628921 7360 raft_consensus.cc:3058] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:33.623412 7473 raft_consensus.cc:2241] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:33.631325 7473 raft_consensus.cc:2270] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:33.634030 7473 ts_tablet_manager.cc:1905] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:33.635560 7360 raft_consensus.cc:2466] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:33.636590 7152 leader_election.cc:304] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:33.587205 6711 catalog_manager.cc:5582] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130). New cstate: current_term: 1 leader_uuid: "315d304854c541bf8b5e6b1eea712d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:33.637382 7609 raft_consensus.cc:2802] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:33.637769 7609 raft_consensus.cc:695] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:33.638494 7609 consensus_queue.cc:237] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.641378 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet b121b5cca8144d9ca9c303709083a101 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d55b03615694e3baa0f4b716a830128) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:33.642100 7559 tablet_replica.cc:331] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:33.642738 7559 raft_consensus.cc:2241] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:33.643401 7559 pending_rounds.cc:70] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca: Trying to abort 1 pending ops.
I20250626 19:58:33.643599 7559 pending_rounds.cc:77] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964574156201984 op_type: NO_OP noop_request { }
I20250626 19:58:33.643893 7559 raft_consensus.cc:2887] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:33.644191 7559 raft_consensus.cc:2270] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:33.647810 7655 leader_election.cc:290] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:33.650355 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b121b5cca8144d9ca9c303709083a101" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
W20250626 19:58:33.651571 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08000c7a60 after lost signal to thread 7129
W20250626 19:58:33.651696 7009 leader_election.cc:343] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: Tablet not RUNNING: STOPPED
W20250626 19:58:33.652616 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08000ca6c0 after lost signal to thread 7260
I20250626 19:58:33.653666 6713 master_service.cc:948] ReplaceTablet: received request to replace tablet db3cbdf7316149f8bf224bafd67461a6 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:33.659142 7673 leader_election.cc:290] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Requested vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:33.673038 7559 ts_tablet_manager.cc:1905] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:33.681295 7569 tablet_bootstrap.cc:654] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:33.682945 7560 ts_tablet_manager.cc:1918] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:33.683310 7560 log.cc:1199] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/b121b5cca8144d9ca9c303709083a101
I20250626 19:58:33.683981 7560 ts_tablet_manager.cc:1939] T b121b5cca8144d9ca9c303709083a101 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:33.687608 7467 tablet_bootstrap.cc:492] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:33.687994 7467 ts_tablet_manager.cc:1397] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.095s user 0.020s sys 0.000s
I20250626 19:58:33.691043 7467 raft_consensus.cc:357] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.691663 7467 raft_consensus.cc:383] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:33.691907 7467 raft_consensus.cc:738] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:33.696889 6716 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_f9f594403d34451f8f2fa6d607d33a2c" } new_table_name: "default.Table_4e311e30e4244e30aa0506381ffc07c0" modify_external_catalogs: true
I20250626 19:58:33.697515 7559 ts_tablet_manager.cc:1918] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:33.697849 7559 log.cc:1199] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/b121b5cca8144d9ca9c303709083a101
I20250626 19:58:33.698484 7559 ts_tablet_manager.cc:1939] T b121b5cca8144d9ca9c303709083a101 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:33.699571 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet b121b5cca8144d9ca9c303709083a101 (table default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]) successfully deleted
I20250626 19:58:33.702011 7473 ts_tablet_manager.cc:1918] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:33.702395 7473 log.cc:1199] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/b121b5cca8144d9ca9c303709083a101
I20250626 19:58:33.703006 7467 consensus_queue.cc:260] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.706897 7467 ts_tablet_manager.cc:1428] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20250626 19:58:33.710610 7702 tablet.cc:1722] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:58:33.712584 7473 ts_tablet_manager.cc:1939] T b121b5cca8144d9ca9c303709083a101 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:33.713383 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet b121b5cca8144d9ca9c303709083a101 (table default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]) successfully deleted
I20250626 19:58:33.719612 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet b121b5cca8144d9ca9c303709083a101 (table default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]) successfully deleted
W20250626 19:58:33.725993 7006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:40373, user_credentials={real_user=slave}} blocked reactor thread for 127995us
I20250626 19:58:33.726720 7679 mvcc.cc:204] Tried to move back new op lower bound from 7171964573975482368 to 7171964570351833088. Current Snapshot: MvccSnapshot[applied={T|T < 7171964573975482368}]
I20250626 19:58:33.728472 7569 tablet_bootstrap.cc:492] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:33.728859 7569 ts_tablet_manager.cc:1397] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.108s user 0.000s sys 0.008s
I20250626 19:58:33.735261 7072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3303f89f34464ad5a39b91293bf11455" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:33.735800 7072 raft_consensus.cc:3058] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:33.737737 7703 tablet.cc:1722] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:58:33.742741 7072 raft_consensus.cc:2466] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:33.749538 6712 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]
I20250626 19:58:33.763171 7569 raft_consensus.cc:357] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.763940 7569 raft_consensus.cc:383] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:33.764230 7569 raft_consensus.cc:738] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:33.771023 7569 consensus_queue.cc:260] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.787309 7569 ts_tablet_manager.cc:1428] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.058s user 0.006s sys 0.000s
I20250626 19:58:33.795574 6713 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet db3cbdf7316149f8bf224bafd67461a6
I20250626 19:58:33.798357 6713 catalog_manager.cc:6652] ReplaceTablet: tablet db3cbdf7316149f8bf224bafd67461a6 deleted and replaced by tablet 7a7b126a6af840adb0c42c5b137620a6
I20250626 19:58:33.798666 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet db3cbdf7316149f8bf224bafd67461a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a7b126a6af840adb0c42c5b137620a6) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:33.799403 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet db3cbdf7316149f8bf224bafd67461a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a7b126a6af840adb0c42c5b137620a6) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:33.800010 7559 tablet_replica.cc:331] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:33.800207 7560 tablet_replica.cc:331] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:33.800767 7559 raft_consensus.cc:2241] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:33.801153 7560 raft_consensus.cc:2241] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:33.801296 7559 raft_consensus.cc:2270] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:33.801946 7560 pending_rounds.cc:70] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a: Trying to abort 1 pending ops.
I20250626 19:58:33.802155 7560 pending_rounds.cc:77] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964573489029120 op_type: NO_OP noop_request { }
I20250626 19:58:33.802476 7560 raft_consensus.cc:2887] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:33.802740 7560 raft_consensus.cc:2270] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:33.803375 7559 ts_tablet_manager.cc:1905] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:33.804992 7560 ts_tablet_manager.cc:1905] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:33.814057 6713 master_service.cc:948] ReplaceTablet: received request to replace tablet 73e216d7176e445dbd2b7a54845636eb from {username='slave'} at 127.0.0.1:42822
W20250626 19:58:33.838042 6704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.189:36417, user_credentials={real_user=slave}} blocked reactor thread for 81400.6us
W20250626 19:58:33.840720 6704 consensus_peers.cc:487] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f -> Peer 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417): Couldn't send request to peer 8ae38ec946e14198ae4adc8dde9d66e8. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250626 19:58:33.862052 7193 tablet_service.cc:1468] Processing CreateTablet for tablet 0d55b03615694e3baa0f4b716a830128 (DEFAULT_TABLE table=default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:33.863087 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d55b03615694e3baa0f4b716a830128. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:33.864434 7341 tablet_service.cc:1468] Processing CreateTablet for tablet 0d55b03615694e3baa0f4b716a830128 (DEFAULT_TABLE table=default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:33.867322 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet b121b5cca8144d9ca9c303709083a101 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d55b03615694e3baa0f4b716a830128) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:33.868839 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet db3cbdf7316149f8bf224bafd67461a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a7b126a6af840adb0c42c5b137620a6) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:33.872227 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 0d55b03615694e3baa0f4b716a830128 (DEFAULT_TABLE table=default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:33.873488 7341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d55b03615694e3baa0f4b716a830128. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:33.874807 7473 tablet_replica.cc:331] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
W20250626 19:58:33.875615 6705 catalog_manager.cc:4895] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet b121b5cca8144d9ca9c303709083a101 because the tablet was not found. No further retry: Not found: Tablet not found: b121b5cca8144d9ca9c303709083a101
I20250626 19:58:33.875962 7473 raft_consensus.cc:2241] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:33.876657 7473 raft_consensus.cc:2270] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:33.877923 7662 raft_consensus.cc:491] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:33.878476 7662 raft_consensus.cc:513] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.881829 7662 leader_election.cc:290] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:33.882865 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73e216d7176e445dbd2b7a54845636eb" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:33.883354 7568 tablet_bootstrap.cc:492] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:33.883441 7215 raft_consensus.cc:2466] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:33.884971 7293 leader_election.cc:304] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:33.897192 7473 ts_tablet_manager.cc:1905] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:33.905289 7560 ts_tablet_manager.cc:1918] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:33.905661 7560 log.cc:1199] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/db3cbdf7316149f8bf224bafd67461a6
I20250626 19:58:33.906351 7560 ts_tablet_manager.cc:1939] T db3cbdf7316149f8bf224bafd67461a6 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:33.907997 7692 raft_consensus.cc:2802] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:33.908454 7692 raft_consensus.cc:491] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:33.908833 7692 raft_consensus.cc:3058] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:33.914546 7568 tablet_bootstrap.cc:654] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:33.922488 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet db3cbdf7316149f8bf224bafd67461a6 (table default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:33.946843 6713 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 73e216d7176e445dbd2b7a54845636eb
I20250626 19:58:33.948226 7568 tablet_bootstrap.cc:492] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:33.948781 7568 ts_tablet_manager.cc:1397] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.066s user 0.011s sys 0.000s
I20250626 19:58:33.949002 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 73e216d7176e445dbd2b7a54845636eb with delete_type TABLET_DATA_DELETED (replaced by tablet edf78b9b864446f08056f25423942db3) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:33.955327 7073 raft_consensus.cc:3058] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:33.958568 7294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.129:33129, user_credentials={real_user=slave}} blocked reactor thread for 93581us
I20250626 19:58:33.960999 7072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73e216d7176e445dbd2b7a54845636eb" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:33.961602 7072 raft_consensus.cc:2466] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:33.964146 6710 catalog_manager.cc:2149] created HMS catalog entry for table [id=def6cb7e52c748fcbe6c2ef2488d5f21]
I20250626 19:58:33.964141 7693 mvcc.cc:204] Tried to move back new op lower bound from 7171964573975482368 to 7171964570351833088. Current Snapshot: MvccSnapshot[applied={T|T < 7171964573975482368}]
I20250626 19:58:33.971324 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet 73e216d7176e445dbd2b7a54845636eb with delete_type TABLET_DATA_DELETED (replaced by tablet edf78b9b864446f08056f25423942db3) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:33.972080 7560 tablet_replica.cc:331] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:33.972741 7560 raft_consensus.cc:2241] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:33.973145 7560 raft_consensus.cc:2270] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:33.975236 7560 ts_tablet_manager.cc:1905] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:33.976966 6944 catalog_manager.cc:2747] Deleting table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f] in response to Hive Metastore notification log event 11
I20250626 19:58:33.984607 7709 tablet.cc:1722] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:58:33.985872 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d55b03615694e3baa0f4b716a830128. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:33.988981 7560 ts_tablet_manager.cc:1918] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:33.989351 7560 log.cc:1199] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/73e216d7176e445dbd2b7a54845636eb
I20250626 19:58:33.954767 7568 raft_consensus.cc:357] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:33.991217 7568 raft_consensus.cc:383] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:33.991467 7568 raft_consensus.cc:738] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:33.992106 7568 consensus_queue.cc:260] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:34.002086 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 73e216d7176e445dbd2b7a54845636eb with delete_type TABLET_DATA_DELETED (replaced by tablet edf78b9b864446f08056f25423942db3) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:34.006479 7692 raft_consensus.cc:513] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:33.989933 7560 ts_tablet_manager.cc:1939] T 73e216d7176e445dbd2b7a54845636eb P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
W20250626 19:58:34.013060 7293 leader_election.cc:343] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: 73e216d7176e445dbd2b7a54845636eb
I20250626 19:58:34.013871 7692 leader_election.cc:290] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:34.013888 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73e216d7176e445dbd2b7a54845636eb" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:34.014477 7071 raft_consensus.cc:3058] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:34.014542 7559 ts_tablet_manager.cc:1918] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:34.015159 7559 log.cc:1199] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/db3cbdf7316149f8bf224bafd67461a6
I20250626 19:58:34.016093 7559 ts_tablet_manager.cc:1939] T db3cbdf7316149f8bf224bafd67461a6 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:34.021720 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet db3cbdf7316149f8bf224bafd67461a6 (table default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:34.023483 6711 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet e8cd3c1cc2de42e8a8503c66811f915f (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:34.027395 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet b121b5cca8144d9ca9c303709083a101 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d55b03615694e3baa0f4b716a830128) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:34.027446 6713 catalog_manager.cc:6652] ReplaceTablet: tablet 73e216d7176e445dbd2b7a54845636eb deleted and replaced by tablet edf78b9b864446f08056f25423942db3
I20250626 19:58:34.032670 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 73e216d7176e445dbd2b7a54845636eb (table default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]) successfully deleted
I20250626 19:58:34.032827 7609 raft_consensus.cc:491] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:34.033231 7609 raft_consensus.cc:513] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:34.035058 7609 leader_election.cc:290] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
W20250626 19:58:34.036541 7152 leader_election.cc:343] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: 0d55b03615694e3baa0f4b716a830128
W20250626 19:58:34.037909 7150 leader_election.cc:343] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Not found: Tablet not found: 0d55b03615694e3baa0f4b716a830128
I20250626 19:58:34.038278 7150 leader_election.cc:304] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a; no voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:34.039422 7609 raft_consensus.cc:2747] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:34.047930 6710 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 14) took 7291 ms. Trace:
I20250626 19:58:34.035858 7568 ts_tablet_manager.cc:1428] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.087s user 0.008s sys 0.000s
I20250626 19:58:34.048369 6710 rpcz_store.cc:276] 0626 19:58:26.756425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:26.756580 (+ 155us) service_pool.cc:225] Handling call
0626 19:58:33.388328 (+6631748us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:33.410921 (+ 22593us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:33.964100 (+553179us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:34.047061 (+ 82961us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:34.047476 (+ 415us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:34.047852 (+ 376us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:33.982633 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:33.983026 (+ 393us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:33.983050 (+ 24us) write_op.cc:435] Acquired schema lock
0626 19:58:33.983061 (+ 11us) tablet.cc:660] Decoding operations
0626 19:58:33.983393 (+ 332us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:33.983433 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:33.983454 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:33.983645 (+ 191us) tablet.cc:699] Row locks acquired
0626 19:58:33.983657 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:58:33.983757 (+ 100us) write_op.cc:270] Start()
0626 19:58:33.983834 (+ 77us) write_op.cc:276] Timestamp: P: 1750967913983725 usec, L: 0
0626 19:58:33.983859 (+ 25us) op_driver.cc:348] REPLICATION: starting
0626 19:58:33.984326 (+ 467us) log.cc:844] Serialized 432 byte log entry
0626 19:58:34.043917 (+ 59591us) op_driver.cc:464] REPLICATION: finished
0626 19:58:34.044350 (+ 433us) write_op.cc:301] APPLY: starting
0626 19:58:34.044449 (+ 99us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:34.044585 (+ 136us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:34.044722 (+ 137us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:34.045219 (+ 497us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:34.045405 (+ 186us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:34.045468 (+ 63us) write_op.cc:312] APPLY: finished
0626 19:58:34.045694 (+ 226us) log.cc:844] Serialized 28 byte log entry
0626 19:58:34.045871 (+ 177us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:34.045964 (+ 93us) write_op.cc:454] Released schema lock
0626 19:58:34.046005 (+ 41us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":336887,"Hive Metastore.run_cpu_time_us":707,"Hive Metastore.run_wall_time_us":215090,"child_traces":[["op",{"apply.queue_time_us":263,"apply.run_cpu_time_us":1874,"apply.run_wall_time_us":1872,"num_ops":2,"prepare.queue_time_us":16564,"raft.queue_time_us":92,"raft.run_cpu_time_us":673,"raft.run_wall_time_us":942,"replication_time_us":59931}]]}
I20250626 19:58:34.075768 7609 consensus_queue.cc:1035] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:34.104166 7655 consensus_queue.cc:1035] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:34.114071 6944 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet caa506927e37438c864d01250977da36
I20250626 19:58:34.116691 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet caa506927e37438c864d01250977da36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:33 UTC) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:34.116894 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet caa506927e37438c864d01250977da36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:33 UTC) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:34.117333 7560 tablet_replica.cc:331] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:34.118028 7560 raft_consensus.cc:2241] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:34.118892 7560 raft_consensus.cc:2270] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:34.122730 7560 ts_tablet_manager.cc:1905] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:34.131783 7569 tablet_bootstrap.cc:492] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:34.132779 7559 tablet_replica.cc:331] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:34.134606 7071 raft_consensus.cc:2466] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:34.137487 7569 tablet_bootstrap.cc:654] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:34.142549 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet e8cd3c1cc2de42e8a8503c66811f915f from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:34.150487 7560 ts_tablet_manager.cc:1918] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:34.150843 7560 log.cc:1199] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/caa506927e37438c864d01250977da36
I20250626 19:58:34.151538 7560 ts_tablet_manager.cc:1939] T caa506927e37438c864d01250977da36 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:34.152158 7294 leader_election.cc:304] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters: 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:34.153623 7692 raft_consensus.cc:2802] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:34.153746 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet caa506927e37438c864d01250977da36 (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
I20250626 19:58:34.155153 7692 raft_consensus.cc:695] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:34.156366 7692 consensus_queue.cc:237] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:34.160847 7559 raft_consensus.cc:2241] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:34.166170 7559 raft_consensus.cc:2270] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 19:58:34.167798 6705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.188:38919, user_credentials={real_user=slave}} blocked reactor thread for 55605.1us
I20250626 19:58:34.169389 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet caa506927e37438c864d01250977da36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:33 UTC) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:34.172008 7569 tablet_bootstrap.cc:492] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:34.172489 7569 ts_tablet_manager.cc:1397] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.001s
I20250626 19:58:34.174593 7569 raft_consensus.cc:357] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:34.175355 7569 raft_consensus.cc:383] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:34.175666 7569 raft_consensus.cc:738] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:34.153172 7473 ts_tablet_manager.cc:1918] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:34.183111 7473 log.cc:1199] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/db3cbdf7316149f8bf224bafd67461a6
I20250626 19:58:34.184180 7473 ts_tablet_manager.cc:1939] T db3cbdf7316149f8bf224bafd67461a6 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:34.191021 7569 consensus_queue.cc:260] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:34.191845 7695 raft_consensus.cc:491] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:34.192251 7695 raft_consensus.cc:513] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:34.193075 7559 ts_tablet_manager.cc:1905] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:34.198866 7695 leader_election.cc:290] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:34.198242 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d55b03615694e3baa0f4b716a830128" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:34.201251 7215 raft_consensus.cc:2466] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:34.203002 7006 leader_election.cc:304] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
W20250626 19:58:34.205001 7009 leader_election.cc:343] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: 0d55b03615694e3baa0f4b716a830128
I20250626 19:58:34.212450 7695 raft_consensus.cc:2802] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:34.213016 7695 raft_consensus.cc:491] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:34.213461 7695 raft_consensus.cc:3058] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:34.248857 7473 tablet_replica.cc:331] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:34.248996 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet db3cbdf7316149f8bf224bafd67461a6 (table default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:34.249750 7473 raft_consensus.cc:2241] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:34.250526 7473 pending_rounds.cc:70] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7: Trying to abort 1 pending ops.
I20250626 19:58:34.250838 7473 pending_rounds.cc:77] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964576522944512 op_type: NO_OP noop_request { }
I20250626 19:58:34.251274 7473 raft_consensus.cc:2887] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:34.251673 7473 raft_consensus.cc:2270] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:34.253931 7695 raft_consensus.cc:513] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:34.260643 7193 tablet_service.cc:1468] Processing CreateTablet for tablet 7a7b126a6af840adb0c42c5b137620a6 (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.261592 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a7b126a6af840adb0c42c5b137620a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:34.262722 7473 ts_tablet_manager.cc:1905] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:34.269995 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 7a7b126a6af840adb0c42c5b137620a6 (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.271230 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a7b126a6af840adb0c42c5b137620a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:34.260043 7569 ts_tablet_manager.cc:1428] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.087s user 0.015s sys 0.011s
I20250626 19:58:34.279273 6718 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42822 (request call id 7) took 8008 ms. Trace:
I20250626 19:58:34.279675 6718 rpcz_store.cc:276] 0626 19:58:26.271228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:26.288857 (+ 17629us) service_pool.cc:225] Handling call
0626 19:58:28.457507 (+2168650us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:34.279148 (+5821641us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":948435,"Hive Metastore.run_cpu_time_us":1134,"Hive Metastore.run_wall_time_us":2898916}
W20250626 19:58:34.280372 6715 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8a5f3c205ce2406e8b57c6f09619cb05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:34.281690 6717 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42822 (request call id 5) took 8042 ms. Trace:
W20250626 19:58:34.284735 6709 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6a693c4bae214626a4050c7a94276f98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:58:34.287773 6714 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_76cdf24ffacc4d129b66c2fd6c0b93d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:34.282085 6717 rpcz_store.cc:276] 0626 19:58:26.238839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:26.238945 (+ 106us) service_pool.cc:225] Handling call
0626 19:58:26.252136 (+ 13191us) spinlock_profiling.cc:243] Waited 5.04 ms on lock 0x7b54000203d4. stack: 00007efd765d2483 00007efd765d2380 00007efd7614e574 00000000004fa174 00000000004fa109 00007efd7d4d4a70 00007efd7d558b1a 00007efd7f205fac 00007efd7f3003b6 00007efd7d025ca4 00007efd7d025c32 00007efd7d025b61 00007efd7d025adc 00007efd7d024d92 00007efd7b318dcc 00007efd7b318206
0626 19:58:26.559329 (+307193us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:34.279523 (+7720194us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:34.280623 (+ 1100us) catalog_manager.cc:3683] Apply alter schema
0626 19:58:34.281536 (+ 913us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":1854739,"Hive Metastore.run_cpu_time_us":1842,"Hive Metastore.run_wall_time_us":991450,"spinlock_wait_cycles":11590912}
I20250626 19:58:34.291582 6717 master_service.cc:948] ReplaceTablet: received request to replace tablet e8cd3c1cc2de42e8a8503c66811f915f from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:34.293864 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d55b03615694e3baa0f4b716a830128" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
W20250626 19:58:34.295296 7009 leader_election.cc:343] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:34.299422 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d55b03615694e3baa0f4b716a830128" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:34.299998 7215 raft_consensus.cc:3058] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:34.309988 7215 raft_consensus.cc:2466] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:34.310220 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 7a7b126a6af840adb0c42c5b137620a6 (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.311434 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a7b126a6af840adb0c42c5b137620a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:34.311342 7006 leader_election.cc:304] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7
I20250626 19:58:34.312275 7715 raft_consensus.cc:2802] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:34.312970 7695 leader_election.cc:290] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Requested vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:34.322342 7715 raft_consensus.cc:695] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Becoming Leader. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Running, Role: LEADER
I20250626 19:58:34.323184 7715 consensus_queue.cc:237] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
W20250626 19:58:34.341552 6703 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 2b9dfd7e091e4d0ea3a698a12a6be585 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2b9dfd7e091e4d0ea3a698a12a6be585 consensus state has no leader
I20250626 19:58:34.345577 7568 tablet_bootstrap.cc:492] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:34.350914 7473 ts_tablet_manager.cc:1918] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:34.351020 7568 tablet_bootstrap.cc:654] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:34.351300 7473 log.cc:1199] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/73e216d7176e445dbd2b7a54845636eb
I20250626 19:58:34.351966 7473 ts_tablet_manager.cc:1939] T 73e216d7176e445dbd2b7a54845636eb P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:34.363209 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 73e216d7176e445dbd2b7a54845636eb (table default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]) successfully deleted
I20250626 19:58:34.367336 7473 tablet_replica.cc:331] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:34.367990 7473 raft_consensus.cc:2241] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:34.368291 7568 tablet_bootstrap.cc:492] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:34.368433 7473 raft_consensus.cc:2270] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:34.370968 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet e8cd3c1cc2de42e8a8503c66811f915f
I20250626 19:58:34.372766 7568 ts_tablet_manager.cc:1397] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.002s
I20250626 19:58:34.375108 7568 raft_consensus.cc:357] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:34.375743 7568 raft_consensus.cc:383] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:34.375995 7473 ts_tablet_manager.cc:1905] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:34.376048 7568 raft_consensus.cc:738] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:34.376677 7568 consensus_queue.cc:260] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:34.394446 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 73e216d7176e445dbd2b7a54845636eb with delete_type TABLET_DATA_DELETED (replaced by tablet edf78b9b864446f08056f25423942db3) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:34.395964 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet e8cd3c1cc2de42e8a8503c66811f915f with delete_type TABLET_DATA_DELETED (replaced by tablet 1abf7ec7ef164277bc0a23338a711027) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:34.402949 7568 ts_tablet_manager.cc:1428] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20250626 19:58:34.403584 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet e8cd3c1cc2de42e8a8503c66811f915f with delete_type TABLET_DATA_DELETED (replaced by tablet 1abf7ec7ef164277bc0a23338a711027) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:34.403932 6715 catalog_manager.cc:2149] created HMS catalog entry for table [id=4f3eec76141f4ec38948625110d7136c]
I20250626 19:58:34.405198 6711 catalog_manager.cc:6652] ReplaceTablet: tablet e8cd3c1cc2de42e8a8503c66811f915f deleted and replaced by tablet 1abf7ec7ef164277bc0a23338a711027
I20250626 19:58:34.405723 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet e8cd3c1cc2de42e8a8503c66811f915f with delete_type TABLET_DATA_DELETED (replaced by tablet 1abf7ec7ef164277bc0a23338a711027) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:34.406391 7560 tablet_replica.cc:331] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:34.373626 6717 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_a43e8ff4d1844c0b9dddf28fc310fd8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:34.407038 7560 raft_consensus.cc:2241] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:34.391974 6713 catalog_manager.cc:5582] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130). New cstate: current_term: 1 leader_uuid: "315d304854c541bf8b5e6b1eea712d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:34.407573 7560 raft_consensus.cc:2270] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:34.408062 6713 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 3303f89f34464ad5a39b91293bf11455 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
I20250626 19:58:34.409829 7560 ts_tablet_manager.cc:1905] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:34.411235 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet e8cd3c1cc2de42e8a8503c66811f915f with delete_type TABLET_DATA_DELETED (replaced by tablet 1abf7ec7ef164277bc0a23338a711027) from {username='slave'} at 127.0.0.1:43274
W20250626 19:58:34.413481 6710 catalog_manager.cc:5415] Ignoring report from unknown tablet caa506927e37438c864d01250977da36
I20250626 19:58:34.417140 7716 tablet_bootstrap.cc:492] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:34.449482 7560 ts_tablet_manager.cc:1918] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:58:34.449831 7560 log.cc:1199] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/e8cd3c1cc2de42e8a8503c66811f915f
I20250626 19:58:34.450486 7560 ts_tablet_manager.cc:1939] T e8cd3c1cc2de42e8a8503c66811f915f P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:34.463876 7473 ts_tablet_manager.cc:1918] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:34.464254 7473 log.cc:1199] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/caa506927e37438c864d01250977da36
I20250626 19:58:34.467903 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet e8cd3c1cc2de42e8a8503c66811f915f (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:34.477722 7473 ts_tablet_manager.cc:1939] T caa506927e37438c864d01250977da36 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:34.481276 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet caa506927e37438c864d01250977da36 (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
I20250626 19:58:34.494773 6715 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 58) took 1414 ms. Trace:
I20250626 19:58:34.495151 6715 rpcz_store.cc:276] 0626 19:58:33.080095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:33.083853 (+ 3758us) service_pool.cc:225] Handling call
0626 19:58:34.280887 (+1197034us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:34.282927 (+ 2040us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:34.403890 (+120963us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:34.494000 (+ 90110us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:34.494332 (+ 332us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:34.494681 (+ 349us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:34.478424 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:34.478933 (+ 509us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:34.478963 (+ 30us) write_op.cc:435] Acquired schema lock
0626 19:58:34.478974 (+ 11us) tablet.cc:660] Decoding operations
0626 19:58:34.479243 (+ 269us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:34.479280 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:34.479302 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:34.479482 (+ 180us) tablet.cc:699] Row locks acquired
0626 19:58:34.479494 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:58:34.479599 (+ 105us) write_op.cc:270] Start()
0626 19:58:34.479691 (+ 92us) write_op.cc:276] Timestamp: P: 1750967914479677 usec, L: 0
0626 19:58:34.479707 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:58:34.480209 (+ 502us) log.cc:844] Serialized 432 byte log entry
0626 19:58:34.492249 (+ 12040us) op_driver.cc:464] REPLICATION: finished
0626 19:58:34.492475 (+ 226us) write_op.cc:301] APPLY: starting
0626 19:58:34.492531 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:34.492642 (+ 111us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:34.492731 (+ 89us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:34.493140 (+ 409us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:34.493272 (+ 132us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:34.493313 (+ 41us) write_op.cc:312] APPLY: finished
0626 19:58:34.493496 (+ 183us) log.cc:844] Serialized 28 byte log entry
0626 19:58:34.493656 (+ 160us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:34.493739 (+ 83us) write_op.cc:454] Released schema lock
0626 19:58:34.493771 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":83,"Hive Metastore.run_cpu_time_us":688,"Hive Metastore.run_wall_time_us":119660,"child_traces":[["op",{"apply.queue_time_us":83,"apply.run_cpu_time_us":1490,"apply.run_wall_time_us":1492,"num_ops":2,"prepare.queue_time_us":72721,"raft.queue_time_us":86,"raft.run_cpu_time_us":659,"raft.run_wall_time_us":660,"replication_time_us":12444}]]}
W20250626 19:58:34.505056 6705 catalog_manager.cc:4895] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 73e216d7176e445dbd2b7a54845636eb because the tablet was not found. No further retry: Not found: Tablet not found: 73e216d7176e445dbd2b7a54845636eb
I20250626 19:58:34.505873 7473 tablet_replica.cc:331] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:34.506492 7473 raft_consensus.cc:2241] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:34.507478 7473 raft_consensus.cc:2270] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:34.509413 7716 tablet_bootstrap.cc:654] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:34.525514 7569 tablet_bootstrap.cc:492] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:34.526028 7559 ts_tablet_manager.cc:1918] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:34.525501 7473 ts_tablet_manager.cc:1905] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:34.534982 6715 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_a5221df6a6b54bcdafa382fca51b613d" } modify_external_catalogs: true
I20250626 19:58:34.535871 6715 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_a5221df6a6b54bcdafa382fca51b613d" } modify_external_catalogs: true
I20250626 19:58:34.537200 6710 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_252448e828024438ba317b8277b66523" } new_table_name: "default.Table_daf65ec234534a069c2d1ad34f13952a" modify_external_catalogs: true
I20250626 19:58:34.530791 7569 tablet_bootstrap.cc:654] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:34.554411 7559 log.cc:1199] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/73e216d7176e445dbd2b7a54845636eb
I20250626 19:58:34.555199 7559 ts_tablet_manager.cc:1939] T 73e216d7176e445dbd2b7a54845636eb P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:34.555919 7716 tablet_bootstrap.cc:492] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:34.557778 7716 ts_tablet_manager.cc:1397] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.141s user 0.007s sys 0.006s
I20250626 19:58:34.557895 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 73e216d7176e445dbd2b7a54845636eb (table default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]) successfully deleted
I20250626 19:58:34.568060 7716 raft_consensus.cc:357] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:34.568524 7569 tablet_bootstrap.cc:492] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:34.568964 7569 ts_tablet_manager.cc:1397] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.002s
I20250626 19:58:34.571295 7569 raft_consensus.cc:357] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:34.571920 7569 raft_consensus.cc:383] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:34.572163 7569 raft_consensus.cc:738] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:34.572853 7569 consensus_queue.cc:260] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:34.575017 7569 ts_tablet_manager.cc:1428] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250626 19:58:34.588125 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet ae04997b77924e8ba479c5c34ac822d7 from {username='slave'} at 127.0.0.1:42822
W20250626 19:58:34.595784 6703 catalog_manager.cc:4895] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet b121b5cca8144d9ca9c303709083a101 because the tablet was not found. No further retry: Not found: Tablet not found: b121b5cca8144d9ca9c303709083a101
I20250626 19:58:34.598237 7559 tablet_replica.cc:331] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:34.604704 7559 raft_consensus.cc:2241] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:34.613283 7559 raft_consensus.cc:2270] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:34.619110 7559 ts_tablet_manager.cc:1905] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:34.627789 6714 catalog_manager.cc:2149] created HMS catalog entry for table [id=ad857025d66f4893872c8b5c92a9c0c7]
I20250626 19:58:34.630764 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 73e216d7176e445dbd2b7a54845636eb with delete_type TABLET_DATA_DELETED (replaced by tablet edf78b9b864446f08056f25423942db3) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:34.643728 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet 3303f89f34464ad5a39b91293bf11455 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:34.653780 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet db3cbdf7316149f8bf224bafd67461a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a7b126a6af840adb0c42c5b137620a6) from {username='slave'} at 127.0.0.1:48028
W20250626 19:58:34.655361 6702 catalog_manager.cc:4895] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet db3cbdf7316149f8bf224bafd67461a6 because the tablet was not found. No further retry: Not found: Tablet not found: db3cbdf7316149f8bf224bafd67461a6
I20250626 19:58:34.657760 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet caa506927e37438c864d01250977da36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:33 UTC) from {username='slave'} at 127.0.0.1:48028
W20250626 19:58:34.659200 6702 catalog_manager.cc:4895] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet caa506927e37438c864d01250977da36 because the tablet was not found. No further retry: Not found: Tablet not found: caa506927e37438c864d01250977da36
I20250626 19:58:34.660964 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet 73e216d7176e445dbd2b7a54845636eb with delete_type TABLET_DATA_DELETED (replaced by tablet edf78b9b864446f08056f25423942db3) from {username='slave'} at 127.0.0.1:48028
W20250626 19:58:34.662233 6702 catalog_manager.cc:4895] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet 73e216d7176e445dbd2b7a54845636eb because the tablet was not found. No further retry: Not found: Tablet not found: 73e216d7176e445dbd2b7a54845636eb
I20250626 19:58:34.662971 7716 raft_consensus.cc:383] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:34.663277 7716 raft_consensus.cc:738] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:34.664109 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet e8cd3c1cc2de42e8a8503c66811f915f with delete_type TABLET_DATA_DELETED (replaced by tablet 1abf7ec7ef164277bc0a23338a711027) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:34.647636 7473 ts_tablet_manager.cc:1918] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:58:34.665038 7473 log.cc:1199] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/e8cd3c1cc2de42e8a8503c66811f915f
W20250626 19:58:34.665513 6702 catalog_manager.cc:4895] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet e8cd3c1cc2de42e8a8503c66811f915f because the tablet was not found. No further retry: Not found: Tablet not found: e8cd3c1cc2de42e8a8503c66811f915f
I20250626 19:58:34.665946 7473 ts_tablet_manager.cc:1939] T e8cd3c1cc2de42e8a8503c66811f915f P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
W20250626 19:58:34.666911 6713 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.130:45539 (request call id 68) took 826 ms (client timeout 1000 ms). Trace:
I20250626 19:58:34.669101 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet e8cd3c1cc2de42e8a8503c66811f915f (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
W20250626 19:58:34.670167 6705 catalog_manager.cc:4895] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet e8cd3c1cc2de42e8a8503c66811f915f because the tablet was not found. No further retry: Not found: Tablet not found: e8cd3c1cc2de42e8a8503c66811f915f
I20250626 19:58:34.663949 7716 consensus_queue.cc:260] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:34.679930 7716 ts_tablet_manager.cc:1428] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.116s user 0.002s sys 0.004s
I20250626 19:58:34.683027 6714 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 61) took 1403 ms. Trace:
I20250626 19:58:34.683343 6714 rpcz_store.cc:276] 0626 19:58:33.279273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:33.281191 (+ 1918us) service_pool.cc:225] Handling call
0626 19:58:34.288187 (+1006996us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:34.289913 (+ 1726us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:34.627743 (+337830us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:34.682338 (+ 54595us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:34.682666 (+ 328us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:34.682970 (+ 304us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:34.629701 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:34.630121 (+ 420us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:34.630166 (+ 45us) write_op.cc:435] Acquired schema lock
0626 19:58:34.630182 (+ 16us) tablet.cc:660] Decoding operations
0626 19:58:34.630464 (+ 282us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:34.630506 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:34.630532 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:34.630773 (+ 241us) tablet.cc:699] Row locks acquired
0626 19:58:34.630790 (+ 17us) write_op.cc:260] PREPARE: finished
0626 19:58:34.630912 (+ 122us) write_op.cc:270] Start()
0626 19:58:34.630992 (+ 80us) write_op.cc:276] Timestamp: P: 1750967914630913 usec, L: 0
0626 19:58:34.631028 (+ 36us) op_driver.cc:348] REPLICATION: starting
0626 19:58:34.631520 (+ 492us) log.cc:844] Serialized 432 byte log entry
0626 19:58:34.652461 (+ 20941us) op_driver.cc:464] REPLICATION: finished
0626 19:58:34.678526 (+ 26065us) write_op.cc:301] APPLY: starting
0626 19:58:34.678601 (+ 75us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:34.678705 (+ 104us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:34.678801 (+ 96us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:34.681546 (+ 2745us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:34.681677 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:34.681714 (+ 37us) write_op.cc:312] APPLY: finished
0626 19:58:34.681880 (+ 166us) log.cc:844] Serialized 28 byte log entry
0626 19:58:34.682021 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:34.682095 (+ 74us) write_op.cc:454] Released schema lock
0626 19:58:34.682131 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":112946,"Hive Metastore.run_cpu_time_us":600,"Hive Metastore.run_wall_time_us":223663,"child_traces":[["op",{"apply.queue_time_us":4643,"num_ops":2,"prepare.queue_time_us":201,"prepare.run_cpu_time_us":2164,"prepare.run_wall_time_us":5563,"raft.queue_time_us":150,"raft.run_cpu_time_us":653,"raft.run_wall_time_us":652,"replication_time_us":21296}]]}
W20250626 19:58:34.667276 6713 rpcz_store.cc:269] 0626 19:58:33.840206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:34.028544 (+188338us) service_pool.cc:225] Handling call
0626 19:58:34.140357 (+111813us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54000203d4. stack: 00007efd765d2483 00007efd765d2380 00007efd7614e574 00000000004fa174 00000000004fa109 00007efd7d4d4a70 00007efd7d558b1a 00007efd7f205fac 00007efd7f2fe7ee 00007efd7d015624 00007efd7d0155b2 00007efd7d0154e1 00007efd7d01545c 00007efd7d014712 00007efd7b318dcc 00007efd7b318206
0626 19:58:34.666854 (+526497us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:34.519391 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:34.519895 (+ 504us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:34.519927 (+ 32us) write_op.cc:435] Acquired schema lock
0626 19:58:34.519947 (+ 20us) tablet.cc:660] Decoding operations
0626 19:58:34.520260 (+ 313us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:34.520313 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:34.520346 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
0626 19:58:34.520533 (+ 187us) tablet.cc:699] Row locks acquired
0626 19:58:34.520552 (+ 19us) write_op.cc:260] PREPARE: finished
0626 19:58:34.520692 (+ 140us) write_op.cc:270] Start()
0626 19:58:34.520795 (+ 103us) write_op.cc:276] Timestamp: P: 1750967914520660 usec, L: 0
0626 19:58:34.520819 (+ 24us) op_driver.cc:348] REPLICATION: starting
0626 19:58:34.539485 (+ 18666us) log.cc:844] Serialized 529 byte log entry
0626 19:58:34.542446 (+ 2961us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b54000203d4. stack: 00007efd765d2483 00007efd765d2380 00007efd7614e574 00000000004fa174 00000000004fa109 00007efd7d4d4481 00007efd7d54ba18 00007efd7d9b8811 00007efd7d9b7fe8 00007efd7d9bcc81 00007efd7d9bcc39 00007efd7d9bcbc9 00007efd7d9bcb91 00007efd7d9bbdbd 000000000059bcd4 000000000059bb09
0626 19:58:34.567981 (+ 25535us) op_driver.cc:464] REPLICATION: finished
0626 19:58:34.568399 (+ 418us) write_op.cc:301] APPLY: starting
0626 19:58:34.568459 (+ 60us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:34.568554 (+ 95us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:34.568569 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:34.568736 (+ 167us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:34.568871 (+ 135us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 19:58:34.568935 (+ 64us) write_op.cc:312] APPLY: finished
0626 19:58:34.569130 (+ 195us) log.cc:844] Serialized 22 byte log entry
0626 19:58:34.569288 (+ 158us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:34.569396 (+ 108us) write_op.cc:454] Released schema lock
0626 19:58:34.569423 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0626 19:58:34.588895 (+ 19472us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:34.588919 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"spinlock_wait_cycles":247938304,"child_traces":[["op",{"apply.queue_time_us":262,"apply.run_cpu_time_us":1199,"apply.run_wall_time_us":1198,"num_ops":1,"prepare.queue_time_us":109278,"prepare.run_cpu_time_us":4099,"prepare.run_wall_time_us":69493,"raft.queue_time_us":16206,"raft.run_cpu_time_us":697,"raft.run_wall_time_us":696,"replication_time_us":47055,"spinlock_wait_cycles":41616128}]]}
I20250626 19:58:34.700353 7716 tablet_bootstrap.cc:492] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:34.706300 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 3303f89f34464ad5a39b91293bf11455
I20250626 19:58:34.707619 6711 catalog_manager.cc:6652] ReplaceTablet: tablet 3303f89f34464ad5a39b91293bf11455 deleted and replaced by tablet 38b99ee757714ee0a3cfe0a22f476d81
I20250626 19:58:34.708391 6711 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_ced1a26248c748fb8c06cd348d12fb19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:34.710006 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet 3303f89f34464ad5a39b91293bf11455 with delete_type TABLET_DATA_DELETED (replaced by tablet 38b99ee757714ee0a3cfe0a22f476d81) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:34.711930 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 3303f89f34464ad5a39b91293bf11455 with delete_type TABLET_DATA_DELETED (replaced by tablet 38b99ee757714ee0a3cfe0a22f476d81) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:34.712266 7560 tablet_replica.cc:331] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:34.713011 7560 raft_consensus.cc:2241] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:34.713634 7560 pending_rounds.cc:70] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a: Trying to abort 1 pending ops.
I20250626 19:58:34.713836 7560 pending_rounds.cc:77] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964574281027584 op_type: NO_OP noop_request { }
I20250626 19:58:34.714135 7560 raft_consensus.cc:2887] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:34.714426 7560 raft_consensus.cc:2270] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:34.717322 7338 tablet_service.cc:1515] Processing DeleteTablet for tablet 3303f89f34464ad5a39b91293bf11455 with delete_type TABLET_DATA_DELETED (replaced by tablet 38b99ee757714ee0a3cfe0a22f476d81) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:34.717947 7473 tablet_replica.cc:331] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:34.718529 7473 raft_consensus.cc:2241] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:34.718971 7473 raft_consensus.cc:2270] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:34.720561 7473 ts_tablet_manager.cc:1905] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:34.721124 7560 ts_tablet_manager.cc:1905] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:34.729715 6718 catalog_manager.cc:5582] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca reported cstate change: term changed from 0 to 1, leader changed from <none> to bbe151be01e24c82854d9c8c79c797ca (127.4.46.129). New cstate: current_term: 1 leader_uuid: "bbe151be01e24c82854d9c8c79c797ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:34.735579 7338 tablet_service.cc:1468] Processing CreateTablet for tablet 17591b5d114f474283965d67d0339153 (DEFAULT_TABLE table=default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.736750 7338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17591b5d114f474283965d67d0339153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:34.738339 7716 tablet_bootstrap.cc:654] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:34.740981 7559 ts_tablet_manager.cc:1918] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:34.741336 7559 log.cc:1199] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/caa506927e37438c864d01250977da36
I20250626 19:58:34.742010 7559 ts_tablet_manager.cc:1939] T caa506927e37438c864d01250977da36 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:34.770473 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet caa506927e37438c864d01250977da36 (table default.table_e1291c46a74d4983aac77cc888bd7962 [id=52b892ca731141a0870ab2cf2864f46f]) successfully deleted
I20250626 19:58:34.772006 7559 tablet_replica.cc:331] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:34.772605 7559 raft_consensus.cc:2241] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:34.773135 7559 raft_consensus.cc:2270] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:34.775655 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 17591b5d114f474283965d67d0339153 (DEFAULT_TABLE table=default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.776551 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17591b5d114f474283965d67d0339153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:34.780764 7193 tablet_service.cc:1468] Processing CreateTablet for tablet 17591b5d114f474283965d67d0339153 (DEFAULT_TABLE table=default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.781666 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17591b5d114f474283965d67d0339153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:34.798990 6713 master_service.cc:948] ReplaceTablet: received request to replace tablet 3303f89f34464ad5a39b91293bf11455 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:34.804311 7051 tablet_service.cc:1468] Processing CreateTablet for tablet edf78b9b864446f08056f25423942db3 (DEFAULT_TABLE table=default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.805063 7569 tablet_bootstrap.cc:492] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:34.805260 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edf78b9b864446f08056f25423942db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:34.810352 7569 tablet_bootstrap.cc:654] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:34.819592 7568 tablet_bootstrap.cc:492] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:34.823861 7196 tablet_service.cc:1468] Processing CreateTablet for tablet edf78b9b864446f08056f25423942db3 (DEFAULT_TABLE table=default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.824865 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edf78b9b864446f08056f25423942db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:34.826453 7559 ts_tablet_manager.cc:1905] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:34.829702 7568 tablet_bootstrap.cc:654] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:34.836849 7716 tablet_bootstrap.cc:492] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
W20250626 19:58:29.011495 7508 debug-util.cc:398] Leaking SignalData structure 0x7b0800012f00 after lost signal to thread 7453
W20250626 19:58:34.839936 7508 kernel_stack_watchdog.cc:198] Thread 7453 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 269ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250626 19:58:27.785270 7509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:58:34.842658 7569 tablet_bootstrap.cc:492] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:34.843145 7569 ts_tablet_manager.cc:1397] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.003s
W20250626 19:58:34.844758 7515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 19:58:34.845345 7569 raft_consensus.cc:357] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:34.847113 7655 raft_consensus.cc:491] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:34.847553 7655 raft_consensus.cc:513] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:34.847707 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 49237fba2f414927b2e18124d5f9bece (DEFAULT_TABLE table=default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.848124 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 49237fba2f414927b2e18124d5f9bece (DEFAULT_TABLE table=default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.848654 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49237fba2f414927b2e18124d5f9bece. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:34.849032 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49237fba2f414927b2e18124d5f9bece. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:34.849295 7655 leader_election.cc:290] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:34.852898 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a7b126a6af840adb0c42c5b137620a6" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:34.853579 7071 raft_consensus.cc:2466] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:34.855083 7150 leader_election.cc:304] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:34.857266 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 1abf7ec7ef164277bc0a23338a711027 (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.858379 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1abf7ec7ef164277bc0a23338a711027. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:34.860101 7195 tablet_service.cc:1468] Processing CreateTablet for tablet 1abf7ec7ef164277bc0a23338a711027 (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.861121 7195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1abf7ec7ef164277bc0a23338a711027. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:34.865500 7568 tablet_bootstrap.cc:492] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:34.865892 7655 raft_consensus.cc:2802] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:34.866237 7655 raft_consensus.cc:491] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 19:58:34.866698 7453 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 7.127s user 0.434s sys 0.940s
W20250626 19:58:34.867065 7453 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 7.127s user 0.434s sys 0.940s
I20250626 19:58:34.838930 7341 tablet_service.cc:1468] Processing CreateTablet for tablet edf78b9b864446f08056f25423942db3 (DEFAULT_TABLE table=default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.866565 7655 raft_consensus.cc:3058] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:34.870816 7341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edf78b9b864446f08056f25423942db3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:58:34.871860 7514 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 7099 milliseconds
I20250626 19:58:34.872421 7453 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 19:58:34.873996 7453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 19:58:34.843559 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 49237fba2f414927b2e18124d5f9bece (DEFAULT_TABLE table=default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.851173 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a7b126a6af840adb0c42c5b137620a6" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:34.875550 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49237fba2f414927b2e18124d5f9bece. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:58:34.876040 7152 leader_election.cc:343] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:34.861057 7340 tablet_service.cc:1468] Processing CreateTablet for tablet 1abf7ec7ef164277bc0a23338a711027 (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:34.877339 7340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1abf7ec7ef164277bc0a23338a711027. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:34.866506 7692 raft_consensus.cc:491] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:34.878954 7692 raft_consensus.cc:513] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:34.880544 7692 leader_election.cc:290] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:34.881928 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d55b03615694e3baa0f4b716a830128" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:34.882572 7215 raft_consensus.cc:2391] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 297d2c913667408bae82668a557e06c7 in current term 1: Already voted for candidate bbe151be01e24c82854d9c8c79c797ca in this term.
I20250626 19:58:34.882967 7473 ts_tablet_manager.cc:1918] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:34.883431 7473 log.cc:1199] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/3303f89f34464ad5a39b91293bf11455
I20250626 19:58:34.884101 7473 ts_tablet_manager.cc:1939] T 3303f89f34464ad5a39b91293bf11455 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:34.885046 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d55b03615694e3baa0f4b716a830128" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:34.886723 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 3303f89f34464ad5a39b91293bf11455 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:34.887719 7294 leader_election.cc:304] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 297d2c913667408bae82668a557e06c7; no voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:34.889022 7692 raft_consensus.cc:3058] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:34.897053 6713 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_6bf87f28d4d34102bf1d02e2e1465a33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:34.898263 7716 ts_tablet_manager.cc:1397] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.200s user 0.006s sys 0.007s
I20250626 19:58:34.845959 7569 raft_consensus.cc:383] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:34.899446 7569 raft_consensus.cc:738] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:34.900072 7569 consensus_queue.cc:260] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:34.902346 7569 ts_tablet_manager.cc:1428] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.059s user 0.004s sys 0.001s
I20250626 19:58:34.900892 7716 raft_consensus.cc:357] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:34.904784 7716 raft_consensus.cc:383] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:34.905054 7716 raft_consensus.cc:738] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:34.905716 7716 consensus_queue.cc:260] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:34.911972 7569 tablet_bootstrap.cc:492] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:34.928718 6714 master_service.cc:948] ReplaceTablet: received request to replace tablet 0d55b03615694e3baa0f4b716a830128 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:34.936460 7559 ts_tablet_manager.cc:1918] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:58:34.936841 7559 log.cc:1199] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/e8cd3c1cc2de42e8a8503c66811f915f
I20250626 19:58:34.937475 7559 ts_tablet_manager.cc:1939] T e8cd3c1cc2de42e8a8503c66811f915f P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:34.918699 7569 tablet_bootstrap.cc:654] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:34.943245 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet e8cd3c1cc2de42e8a8503c66811f915f (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
W20250626 19:58:34.944857 6703 catalog_manager.cc:4895] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 73e216d7176e445dbd2b7a54845636eb because the tablet was not found. No further retry: Not found: Tablet not found: 73e216d7176e445dbd2b7a54845636eb
I20250626 19:58:34.865900 7568 ts_tablet_manager.cc:1397] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.001s
I20250626 19:58:34.945638 7559 tablet_replica.cc:331] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:34.946251 7559 raft_consensus.cc:2241] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:34.946635 7559 raft_consensus.cc:2270] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:34.948103 7559 ts_tablet_manager.cc:1905] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:34.949810 6714 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 0d55b03615694e3baa0f4b716a830128
I20250626 19:58:34.950841 7560 ts_tablet_manager.cc:1918] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:34.951280 6714 catalog_manager.cc:6652] ReplaceTablet: tablet 0d55b03615694e3baa0f4b716a830128 deleted and replaced by tablet 73d091b5175d49398beb3ea231a93f63
I20250626 19:58:34.952032 7560 log.cc:1199] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/3303f89f34464ad5a39b91293bf11455
I20250626 19:58:34.952587 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d55b03615694e3baa0f4b716a830128 with delete_type TABLET_DATA_DELETED (replaced by tablet 73d091b5175d49398beb3ea231a93f63) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:34.952754 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d55b03615694e3baa0f4b716a830128 with delete_type TABLET_DATA_DELETED (replaced by tablet 73d091b5175d49398beb3ea231a93f63) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:34.952899 7337 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d55b03615694e3baa0f4b716a830128 with delete_type TABLET_DATA_DELETED (replaced by tablet 73d091b5175d49398beb3ea231a93f63) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:34.953544 7473 tablet_replica.cc:331] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:34.955809 7569 tablet_bootstrap.cc:492] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:34.956198 7569 ts_tablet_manager.cc:1397] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.000s
I20250626 19:58:34.957661 7655 raft_consensus.cc:513] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:34.958390 7569 raft_consensus.cc:357] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:34.959090 7569 raft_consensus.cc:383] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:34.959285 7453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 19:58:34.959340 7569 raft_consensus.cc:738] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:34.959785 7568 raft_consensus.cc:357] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:34.959980 7569 consensus_queue.cc:260] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:34.960443 7568 raft_consensus.cc:383] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:34.960687 7568 raft_consensus.cc:738] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:34.964927 7560 ts_tablet_manager.cc:1939] T 3303f89f34464ad5a39b91293bf11455 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:34.965484 7569 ts_tablet_manager.cc:1428] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250626 19:58:34.971750 7569 tablet_bootstrap.cc:492] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:34.975772 7453 hybrid_clock.cc:648] HybridClock initialized: now 1750967914975701 us; error 65 us; skew 500 ppm
I20250626 19:58:34.977046 7453 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 19:58:34.979096 7692 raft_consensus.cc:2747] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:34.980671 7569 tablet_bootstrap.cc:654] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:34.982344 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 3303f89f34464ad5a39b91293bf11455 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:34.983286 7560 tablet_replica.cc:331] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:34.984035 7560 raft_consensus.cc:2241] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:34.984529 7560 raft_consensus.cc:2270] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:34.986097 7560 ts_tablet_manager.cc:1905] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:34.987212 7559 ts_tablet_manager.cc:1918] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:34.987571 7559 log.cc:1199] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/3303f89f34464ad5a39b91293bf11455
I20250626 19:58:34.988153 7559 ts_tablet_manager.cc:1939] T 3303f89f34464ad5a39b91293bf11455 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:34.990307 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 3303f89f34464ad5a39b91293bf11455 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:34.991376 7473 raft_consensus.cc:2241] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:34.991958 7473 raft_consensus.cc:2270] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:34.994040 7473 ts_tablet_manager.cc:1905] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:34.994205 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a7b126a6af840adb0c42c5b137620a6" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:34.994503 7716 ts_tablet_manager.cc:1428] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.096s user 0.006s sys 0.000s
I20250626 19:58:34.994787 7071 raft_consensus.cc:3058] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:34.994796 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a7b126a6af840adb0c42c5b137620a6" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:34.995325 7360 raft_consensus.cc:3058] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:34.996134 7716 tablet_bootstrap.cc:492] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:35.001899 7360 raft_consensus.cc:2466] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:35.006785 7071 raft_consensus.cc:2466] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:35.010080 7473 ts_tablet_manager.cc:1918] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:35.010413 7473 log.cc:1199] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/0d55b03615694e3baa0f4b716a830128
I20250626 19:58:35.011098 7473 ts_tablet_manager.cc:1939] T 0d55b03615694e3baa0f4b716a830128 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:35.017472 6714 master_service.cc:948] ReplaceTablet: received request to replace tablet ae04997b77924e8ba479c5c34ac822d7 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:35.018144 7559 tablet_replica.cc:331] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:35.018836 7559 raft_consensus.cc:2241] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:35.019457 7559 pending_rounds.cc:70] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca: Trying to abort 1 pending ops.
I20250626 19:58:35.019706 7559 pending_rounds.cc:77] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964577086562304 op_type: NO_OP noop_request { }
I20250626 19:58:35.020036 7559 raft_consensus.cc:2887] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:35.020350 7559 raft_consensus.cc:2270] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:35.022078 7569 tablet_bootstrap.cc:492] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:35.022459 7569 ts_tablet_manager.cc:1397] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.000s
I20250626 19:58:35.024636 7569 raft_consensus.cc:357] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.025278 7569 raft_consensus.cc:383] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.025542 7569 raft_consensus.cc:738] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.026269 7569 consensus_queue.cc:260] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.030046 7569 ts_tablet_manager.cc:1428] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250626 19:58:35.031690 7559 ts_tablet_manager.cc:1905] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:34.961205 7568 consensus_queue.cc:260] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.037251 7569 tablet_bootstrap.cc:492] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:35.003057 7152 leader_election.cc:304] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:35.008713 7655 leader_election.cc:290] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:35.038975 7655 raft_consensus.cc:2802] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:35.039386 7655 raft_consensus.cc:695] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:35.040102 7655 consensus_queue.cc:237] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.042992 7569 tablet_bootstrap.cc:654] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.044853 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 0d55b03615694e3baa0f4b716a830128 (table default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]) successfully deleted
I20250626 19:58:35.051385 7568 ts_tablet_manager.cc:1428] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.105s user 0.004s sys 0.001s
I20250626 19:58:35.052261 7568 tablet_bootstrap.cc:492] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:35.054018 7559 ts_tablet_manager.cc:1918] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:35.054409 7559 log.cc:1199] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/0d55b03615694e3baa0f4b716a830128
I20250626 19:58:35.055101 7559 ts_tablet_manager.cc:1939] T 0d55b03615694e3baa0f4b716a830128 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:35.063196 7568 tablet_bootstrap.cc:654] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.063585 7569 tablet_bootstrap.cc:492] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:35.064074 7569 ts_tablet_manager.cc:1397] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.002s
I20250626 19:58:35.066605 7569 raft_consensus.cc:357] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.067211 7569 raft_consensus.cc:383] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.067461 7569 raft_consensus.cc:738] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.067672 7453 webserver.cc:469] Webserver started at http://127.4.46.189:37313/ using document root <none> and password file <none>
I20250626 19:58:35.069057 7453 fs_manager.cc:362] Metadata directory not provided
I20250626 19:58:35.069415 7453 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 19:58:35.070114 7569 consensus_queue.cc:260] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.076197 7569 ts_tablet_manager.cc:1428] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.012s user 0.005s sys 0.002s
I20250626 19:58:35.079841 7716 tablet_bootstrap.cc:654] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:35.080482 7152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.131:33703, user_credentials={real_user=slave}} blocked reactor thread for 77534.4us
I20250626 19:58:35.069557 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 0d55b03615694e3baa0f4b716a830128 (table default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]) successfully deleted
I20250626 19:58:35.115880 7568 tablet_bootstrap.cc:492] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:35.116262 7568 ts_tablet_manager.cc:1397] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.004s
I20250626 19:58:35.118289 7568 raft_consensus.cc:357] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.118930 7568 raft_consensus.cc:383] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.119163 7568 raft_consensus.cc:738] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.120074 7568 consensus_queue.cc:260] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.120478 7560 ts_tablet_manager.cc:1918] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:35.120791 7560 log.cc:1199] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/0d55b03615694e3baa0f4b716a830128
I20250626 19:58:35.128285 6709 catalog_manager.cc:2149] created HMS catalog entry for table [id=7c0b8a662d1442b5b758ace1e136d79a]
I20250626 19:58:35.131090 7568 ts_tablet_manager.cc:1428] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250626 19:58:35.133009 7560 ts_tablet_manager.cc:1939] T 0d55b03615694e3baa0f4b716a830128 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:35.137212 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 0d55b03615694e3baa0f4b716a830128 (table default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]) successfully deleted
I20250626 19:58:35.139811 7568 tablet_bootstrap.cc:492] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:35.157241 7340 tablet_service.cc:1468] Processing CreateTablet for tablet 9ee6604fe87a43398a9545ea70ce19c4 (DEFAULT_TABLE table=default.table_76cdf24ffacc4d129b66c2fd6c0b93d8 [id=ad857025d66f4893872c8b5c92a9c0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.157850 7341 tablet_service.cc:1468] Processing CreateTablet for tablet 73d091b5175d49398beb3ea231a93f63 (DEFAULT_TABLE table=default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.158340 7340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee6604fe87a43398a9545ea70ce19c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.161532 7341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73d091b5175d49398beb3ea231a93f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.164882 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 73d091b5175d49398beb3ea231a93f63 (DEFAULT_TABLE table=default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.165113 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 9ee6604fe87a43398a9545ea70ce19c4 (DEFAULT_TABLE table=default.table_76cdf24ffacc4d129b66c2fd6c0b93d8 [id=ad857025d66f4893872c8b5c92a9c0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.165434 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 38b99ee757714ee0a3cfe0a22f476d81 (DEFAULT_TABLE table=default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.165961 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73d091b5175d49398beb3ea231a93f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.167901 7195 tablet_service.cc:1468] Processing CreateTablet for tablet 73d091b5175d49398beb3ea231a93f63 (DEFAULT_TABLE table=default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.168586 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee6604fe87a43398a9545ea70ce19c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.168911 7195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73d091b5175d49398beb3ea231a93f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.168948 6714 master_service.cc:948] ReplaceTablet: received request to replace tablet 49237fba2f414927b2e18124d5f9bece from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:35.171365 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 9ee6604fe87a43398a9545ea70ce19c4 (DEFAULT_TABLE table=default.table_76cdf24ffacc4d129b66c2fd6c0b93d8 [id=ad857025d66f4893872c8b5c92a9c0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.172292 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee6604fe87a43398a9545ea70ce19c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.172271 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b99ee757714ee0a3cfe0a22f476d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.180686 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 38b99ee757714ee0a3cfe0a22f476d81 (DEFAULT_TABLE table=default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.181636 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b99ee757714ee0a3cfe0a22f476d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.182121 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 38b99ee757714ee0a3cfe0a22f476d81 (DEFAULT_TABLE table=default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.183108 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b99ee757714ee0a3cfe0a22f476d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.185863 6709 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 54) took 2514 ms. Trace:
I20250626 19:58:35.188634 7715 raft_consensus.cc:491] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:35.186164 6709 rpcz_store.cc:276] 0626 19:58:32.671454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:32.674934 (+ 3480us) service_pool.cc:225] Handling call
0626 19:58:34.285121 (+1610187us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:34.286702 (+ 1581us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:35.128243 (+841541us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:35.185270 (+ 57027us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:35.185555 (+ 285us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:35.185803 (+ 248us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:35.129899 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:35.130262 (+ 363us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:35.130284 (+ 22us) write_op.cc:435] Acquired schema lock
0626 19:58:35.130296 (+ 12us) tablet.cc:660] Decoding operations
0626 19:58:35.130549 (+ 253us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:35.130582 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:35.130603 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:35.130816 (+ 213us) tablet.cc:699] Row locks acquired
0626 19:58:35.130829 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:58:35.130934 (+ 105us) write_op.cc:270] Start()
0626 19:58:35.131017 (+ 83us) write_op.cc:276] Timestamp: P: 1750967915130920 usec, L: 0
0626 19:58:35.131037 (+ 20us) op_driver.cc:348] REPLICATION: starting
0626 19:58:35.131599 (+ 562us) log.cc:844] Serialized 432 byte log entry
0626 19:58:35.183232 (+ 51633us) op_driver.cc:464] REPLICATION: finished
0626 19:58:35.183545 (+ 313us) write_op.cc:301] APPLY: starting
0626 19:58:35.183604 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:35.183712 (+ 108us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:35.183791 (+ 79us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:35.184190 (+ 399us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:35.184318 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:35.184363 (+ 45us) write_op.cc:312] APPLY: finished
0626 19:58:35.184556 (+ 193us) log.cc:844] Serialized 28 byte log entry
0626 19:58:35.184709 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:35.184771 (+ 62us) write_op.cc:454] Released schema lock
0626 19:58:35.184807 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0626 19:58:35.185013 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:35.185034 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":327858,"Hive Metastore.run_cpu_time_us":645,"Hive Metastore.run_wall_time_us":501022,"child_traces":[["op",{"apply.queue_time_us":167,"apply.run_cpu_time_us":1439,"apply.run_wall_time_us":1439,"num_ops":2,"prepare.queue_time_us":194,"prepare.run_cpu_time_us":1787,"prepare.run_wall_time_us":1884,"replication_time_us":52090}]]}
I20250626 19:58:35.189069 7715 raft_consensus.cc:513] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.190855 7715 leader_election.cc:290] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:35.192849 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17591b5d114f474283965d67d0339153" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
W20250626 19:58:35.194226 7009 leader_election.cc:343] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.198051 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17591b5d114f474283965d67d0339153" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:35.198647 7215 raft_consensus.cc:2466] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:35.199597 7006 leader_election.cc:304] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7
I20250626 19:58:35.200690 7715 raft_consensus.cc:2802] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:35.201011 7715 raft_consensus.cc:491] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:35.201308 7715 raft_consensus.cc:3058] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:35.212456 7733 raft_consensus.cc:491] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:35.212905 7733 raft_consensus.cc:513] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.213770 7715 raft_consensus.cc:513] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.214718 7733 leader_election.cc:290] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:35.217180 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49237fba2f414927b2e18124d5f9bece" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:35.218245 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17591b5d114f474283965d67d0339153" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
W20250626 19:58:35.218374 7152 leader_election.cc:343] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.220299 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49237fba2f414927b2e18124d5f9bece" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:35.220865 7071 raft_consensus.cc:2466] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
W20250626 19:58:35.222543 7009 leader_election.cc:343] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.224239 7150 leader_election.cc:304] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7
I20250626 19:58:35.225389 7733 raft_consensus.cc:2802] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:35.225687 7733 raft_consensus.cc:491] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:35.225975 7733 raft_consensus.cc:3058] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:35.227764 7568 tablet_bootstrap.cc:654] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.236987 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 7a7b126a6af840adb0c42c5b137620a6 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:35.244616 7733 raft_consensus.cc:513] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.247627 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49237fba2f414927b2e18124d5f9bece" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:35.247710 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49237fba2f414927b2e18124d5f9bece" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:35.248255 7071 raft_consensus.cc:3058] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:35.248908 7152 leader_election.cc:343] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.250039 6714 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 49237fba2f414927b2e18124d5f9bece
I20250626 19:58:35.254693 7071 raft_consensus.cc:2466] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:35.255750 7150 leader_election.cc:304] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7
I20250626 19:58:35.256456 7655 raft_consensus.cc:2802] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:35.256868 7655 raft_consensus.cc:695] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:35.257066 6714 catalog_manager.cc:6652] ReplaceTablet: tablet 49237fba2f414927b2e18124d5f9bece deleted and replaced by tablet c91a8c1915eb40508b2c1482e03d6638
I20250626 19:58:35.257591 7655 consensus_queue.cc:237] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.261039 7053 tablet_service.cc:1515] Processing DeleteTablet for tablet 49237fba2f414927b2e18124d5f9bece with delete_type TABLET_DATA_DELETED (replaced by tablet c91a8c1915eb40508b2c1482e03d6638) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:35.261785 7559 tablet_replica.cc:331] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:35.262676 7559 raft_consensus.cc:2241] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:35.262249 7742 raft_consensus.cc:491] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:35.263523 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 49237fba2f414927b2e18124d5f9bece with delete_type TABLET_DATA_DELETED (replaced by tablet c91a8c1915eb40508b2c1482e03d6638) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:35.269505 7560 tablet_replica.cc:331] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:35.280532 7568 tablet_bootstrap.cc:492] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:35.281157 7568 ts_tablet_manager.cc:1397] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.142s user 0.006s sys 0.004s
I20250626 19:58:35.283380 7568 raft_consensus.cc:357] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.263267 7559 raft_consensus.cc:2270] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:35.284159 7742 raft_consensus.cc:513] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.285979 7559 ts_tablet_manager.cc:1905] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:35.288672 7742 leader_election.cc:290] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:35.289093 7568 raft_consensus.cc:383] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.289407 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1abf7ec7ef164277bc0a23338a711027" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
W20250626 19:58:35.291980 7006 leader_election.cc:343] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.289408 7568 raft_consensus.cc:738] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.293407 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1abf7ec7ef164277bc0a23338a711027" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:35.294051 7568 consensus_queue.cc:260] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:35.294744 7009 leader_election.cc:343] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.296306 7568 ts_tablet_manager.cc:1428] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20250626 19:58:35.297111 7568 tablet_bootstrap.cc:492] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:35.297351 7569 tablet_bootstrap.cc:492] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:35.298158 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 49237fba2f414927b2e18124d5f9bece with delete_type TABLET_DATA_DELETED (replaced by tablet c91a8c1915eb40508b2c1482e03d6638) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:35.298910 7009 leader_election.cc:304] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:35.303911 7569 tablet_bootstrap.cc:654] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:35.306921 6705 catalog_manager.cc:4908] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 49237fba2f414927b2e18124d5f9bece because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49237fba2f414927b2e18124d5f9bece already in progress: creating tablet
I20250626 19:58:35.308534 7716 tablet_bootstrap.cc:492] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:35.308956 7716 ts_tablet_manager.cc:1397] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.313s user 0.009s sys 0.004s
I20250626 19:58:35.310940 7560 raft_consensus.cc:2241] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:35.312125 7560 pending_rounds.cc:70] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a: Trying to abort 1 pending ops.
I20250626 19:58:35.312327 7560 pending_rounds.cc:77] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964580909010944 op_type: NO_OP noop_request { }
I20250626 19:58:35.312629 7560 raft_consensus.cc:2887] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:35.312858 7453 fs_manager.cc:714] Time spent opening directory manager: real 0.238s user 0.005s sys 0.017s
I20250626 19:58:35.312938 7560 raft_consensus.cc:2270] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:35.319566 7747 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 19:58:35.319733 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17591b5d114f474283965d67d0339153" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:35.320312 7215 raft_consensus.cc:3058] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:35.323063 6712 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42822 (request call id 11) took 8751 ms. Trace:
I20250626 19:58:35.324712 7569 tablet_bootstrap.cc:492] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:35.325102 7569 ts_tablet_manager.cc:1397] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.000s
I20250626 19:58:35.328193 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 7a7b126a6af840adb0c42c5b137620a6
I20250626 19:58:35.328120 7569 raft_consensus.cc:357] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.328743 7569 raft_consensus.cc:383] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.329005 7742 raft_consensus.cc:2747] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:35.329569 7215 raft_consensus.cc:2466] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:35.330389 7559 ts_tablet_manager.cc:1918] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:35.330739 7559 log.cc:1199] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/49237fba2f414927b2e18124d5f9bece
I20250626 19:58:35.329006 7569 raft_consensus.cc:738] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.330967 7006 leader_election.cc:304] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7
I20250626 19:58:35.331480 7569 consensus_queue.cc:260] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.331873 7733 leader_election.cc:290] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:35.338857 7715 leader_election.cc:290] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Requested vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:35.339511 7559 ts_tablet_manager.cc:1939] T 49237fba2f414927b2e18124d5f9bece P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:35.323382 6712 rpcz_store.cc:276] 0626 19:58:26.571741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:26.571877 (+ 136us) service_pool.cc:225] Handling call
0626 19:58:33.386489 (+6814612us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:35.323000 (+1936511us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":106,"Hive Metastore.run_cpu_time_us":358,"Hive Metastore.run_wall_time_us":360834}
I20250626 19:58:35.341807 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a7b126a6af840adb0c42c5b137620a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 611d4317ec3249819fd98b7d59da87d6) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:35.343350 7569 ts_tablet_manager.cc:1428] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.018s user 0.007s sys 0.001s
I20250626 19:58:35.343495 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 7a7b126a6af840adb0c42c5b137620a6 deleted and replaced by tablet 611d4317ec3249819fd98b7d59da87d6
I20250626 19:58:35.343632 7340 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a7b126a6af840adb0c42c5b137620a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 611d4317ec3249819fd98b7d59da87d6) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:35.344343 7473 tablet_replica.cc:331] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:35.338909 7742 raft_consensus.cc:2802] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:35.344982 7473 raft_consensus.cc:2241] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:35.345173 7742 raft_consensus.cc:695] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Becoming Leader. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Running, Role: LEADER
I20250626 19:58:35.345410 7473 raft_consensus.cc:2270] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:35.345924 7742 consensus_queue.cc:237] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.349252 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 49237fba2f414927b2e18124d5f9bece (table default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]) successfully deleted
W20250626 19:58:35.349790 6704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.189:36417, user_credentials={real_user=slave}} blocked reactor thread for 243454us
I20250626 19:58:35.350657 7569 tablet_bootstrap.cc:492] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:35.351498 7473 ts_tablet_manager.cc:1905] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:35.356753 7569 tablet_bootstrap.cc:654] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.363382 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a7b126a6af840adb0c42c5b137620a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 611d4317ec3249819fd98b7d59da87d6) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:35.364331 7559 tablet_replica.cc:331] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:35.365020 7559 raft_consensus.cc:2241] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:35.365460 7559 raft_consensus.cc:2270] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:35.367174 7559 ts_tablet_manager.cc:1905] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:35.367578 7560 ts_tablet_manager.cc:1905] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:35.369817 7568 tablet_bootstrap.cc:654] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.372680 7453 fs_manager.cc:730] Time spent opening block manager: real 0.057s user 0.001s sys 0.003s
I20250626 19:58:35.373129 7453 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data,/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/wal
uuid: "8ae38ec946e14198ae4adc8dde9d66e8"
format_stamp: "Formatted at 2025-06-26 19:58:14 on dist-test-slave-48dm"
I20250626 19:58:35.375950 7453 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 19:58:35.379240 7716 raft_consensus.cc:357] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.380089 7716 raft_consensus.cc:383] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.380388 7716 raft_consensus.cc:738] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.381037 7716 consensus_queue.cc:260] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.396116 6712 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_fa425acbc3844433a95ac1c8ee1dc26d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:35.403730 7569 tablet_bootstrap.cc:492] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:35.404157 7569 ts_tablet_manager.cc:1397] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.005s
I20250626 19:58:35.408931 7559 ts_tablet_manager.cc:1918] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:35.409312 7559 log.cc:1199] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/7a7b126a6af840adb0c42c5b137620a6
I20250626 19:58:35.409970 7559 ts_tablet_manager.cc:1939] T 7a7b126a6af840adb0c42c5b137620a6 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:35.412359 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 7a7b126a6af840adb0c42c5b137620a6 (table default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:35.406454 7569 raft_consensus.cc:357] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.417491 7569 raft_consensus.cc:383] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.417796 7569 raft_consensus.cc:738] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.418520 7569 consensus_queue.cc:260] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.419268 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 101f870043e44be7b3bf098c37b108af (DEFAULT_TABLE table=default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.420281 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 101f870043e44be7b3bf098c37b108af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.422438 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 101f870043e44be7b3bf098c37b108af (DEFAULT_TABLE table=default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.423426 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 101f870043e44be7b3bf098c37b108af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.424292 7340 tablet_service.cc:1468] Processing CreateTablet for tablet c91a8c1915eb40508b2c1482e03d6638 (DEFAULT_TABLE table=default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.425657 7340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91a8c1915eb40508b2c1482e03d6638. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.422952 7569 ts_tablet_manager.cc:1428] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20250626 19:58:35.431545 7341 tablet_service.cc:1468] Processing CreateTablet for tablet 101f870043e44be7b3bf098c37b108af (DEFAULT_TABLE table=default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.432792 7341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 101f870043e44be7b3bf098c37b108af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.435006 7569 tablet_bootstrap.cc:492] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:35.437875 7050 tablet_service.cc:1468] Processing CreateTablet for tablet c91a8c1915eb40508b2c1482e03d6638 (DEFAULT_TABLE table=default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.439011 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91a8c1915eb40508b2c1482e03d6638. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.445111 7569 tablet_bootstrap.cc:654] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.448300 7194 tablet_service.cc:1468] Processing CreateTablet for tablet c91a8c1915eb40508b2c1482e03d6638 (DEFAULT_TABLE table=default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.449162 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91a8c1915eb40508b2c1482e03d6638. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.466986 7716 ts_tablet_manager.cc:1428] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.158s user 0.007s sys 0.000s
I20250626 19:58:35.470337 7716 tablet_bootstrap.cc:492] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:35.475791 6718 master_service.cc:948] ReplaceTablet: received request to replace tablet 1abf7ec7ef164277bc0a23338a711027 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:35.476311 7569 tablet_bootstrap.cc:492] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:35.476699 7569 ts_tablet_manager.cc:1397] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.000s
I20250626 19:58:35.479353 7569 raft_consensus.cc:357] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:35.479940 7569 raft_consensus.cc:383] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.480571 7568 tablet_bootstrap.cc:492] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:35.481009 7568 ts_tablet_manager.cc:1397] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.184s user 0.005s sys 0.004s
I20250626 19:58:35.481144 7569 raft_consensus.cc:738] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.481810 7569 consensus_queue.cc:260] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:35.483460 7568 raft_consensus.cc:357] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.484369 7569 ts_tablet_manager.cc:1428] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20250626 19:58:35.484148 7568 raft_consensus.cc:383] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.485504 7568 raft_consensus.cc:738] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.486197 7568 consensus_queue.cc:260] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.489289 7742 raft_consensus.cc:491] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:35.489686 7742 raft_consensus.cc:513] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.491480 7742 leader_election.cc:290] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:35.492594 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73d091b5175d49398beb3ea231a93f63" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
W20250626 19:58:35.495012 7009 leader_election.cc:343] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:58:35.517590 6702 catalog_manager.cc:5001] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): alter failed for tablet 3303f89f34464ad5a39b91293bf11455 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]),no further retry: Not found: Tablet not found: 3303f89f34464ad5a39b91293bf11455
I20250626 19:58:35.526413 7716 tablet_bootstrap.cc:654] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.527073 7473 ts_tablet_manager.cc:1918] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:35.527420 7473 log.cc:1199] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/7a7b126a6af840adb0c42c5b137620a6
I20250626 19:58:35.527977 7473 ts_tablet_manager.cc:1939] T 7a7b126a6af840adb0c42c5b137620a6 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:35.492233 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73d091b5175d49398beb3ea231a93f63" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:35.488485 7568 ts_tablet_manager.cc:1428] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
W20250626 19:58:35.532198 7006 leader_election.cc:343] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.532619 7006 leader_election.cc:304] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:35.533277 7568 tablet_bootstrap.cc:492] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:35.533631 7742 raft_consensus.cc:2747] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:35.539124 7560 ts_tablet_manager.cc:1918] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:35.539464 7560 log.cc:1199] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/49237fba2f414927b2e18124d5f9bece
I20250626 19:58:35.540073 7560 ts_tablet_manager.cc:1939] T 49237fba2f414927b2e18124d5f9bece P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:35.542233 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 49237fba2f414927b2e18124d5f9bece (table default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]) successfully deleted
I20250626 19:58:35.549620 7569 tablet_bootstrap.cc:492] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:35.558048 7569 tablet_bootstrap.cc:654] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.577298 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 611d4317ec3249819fd98b7d59da87d6 (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.578378 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 611d4317ec3249819fd98b7d59da87d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.578718 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 611d4317ec3249819fd98b7d59da87d6 (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.579694 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 611d4317ec3249819fd98b7d59da87d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.594514 7568 tablet_bootstrap.cc:654] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.596810 6718 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 1abf7ec7ef164277bc0a23338a711027
I20250626 19:58:35.598176 6718 catalog_manager.cc:6652] ReplaceTablet: tablet 1abf7ec7ef164277bc0a23338a711027 deleted and replaced by tablet 0c02e3d2b20b4577b00d131cf08643bd
I20250626 19:58:35.599983 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 1abf7ec7ef164277bc0a23338a711027 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c02e3d2b20b4577b00d131cf08643bd) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:35.600781 7559 tablet_replica.cc:331] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:35.601469 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 1abf7ec7ef164277bc0a23338a711027 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c02e3d2b20b4577b00d131cf08643bd) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:35.601528 7559 raft_consensus.cc:2241] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:35.601938 7559 raft_consensus.cc:2270] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:35.603909 7559 ts_tablet_manager.cc:1905] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:35.609158 6714 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 38b99ee757714ee0a3cfe0a22f476d81 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
I20250626 19:58:35.611579 6714 catalog_manager.cc:5582] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca reported cstate change: term changed from 0 to 1, leader changed from <none> to bbe151be01e24c82854d9c8c79c797ca (127.4.46.129). New cstate: current_term: 1 leader_uuid: "bbe151be01e24c82854d9c8c79c797ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:35.609884 6718 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_8a5f3c205ce2406e8b57c6f09619cb05" } modify_external_catalogs: true
I20250626 19:58:35.621282 6718 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_8a5f3c205ce2406e8b57c6f09619cb05" } modify_external_catalogs: true
I20250626 19:58:35.627725 7742 raft_consensus.cc:491] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:35.628219 7742 raft_consensus.cc:513] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.630187 7742 leader_election.cc:290] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:35.632297 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38b99ee757714ee0a3cfe0a22f476d81" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
W20250626 19:58:35.633942 7006 leader_election.cc:343] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.637634 7559 ts_tablet_manager.cc:1918] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:35.638047 7559 log.cc:1199] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/1abf7ec7ef164277bc0a23338a711027
I20250626 19:58:35.652973 7568 tablet_bootstrap.cc:492] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:35.653625 7568 ts_tablet_manager.cc:1397] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.121s user 0.010s sys 0.000s
I20250626 19:58:35.656616 7568 raft_consensus.cc:357] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.660194 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 1abf7ec7ef164277bc0a23338a711027 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:35.665978 7559 ts_tablet_manager.cc:1939] T 1abf7ec7ef164277bc0a23338a711027 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:35.668341 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 1abf7ec7ef164277bc0a23338a711027 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:35.674847 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 1abf7ec7ef164277bc0a23338a711027 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c02e3d2b20b4577b00d131cf08643bd) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:35.675691 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 1abf7ec7ef164277bc0a23338a711027 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c02e3d2b20b4577b00d131cf08643bd) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:35.676456 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 7a7b126a6af840adb0c42c5b137620a6 (table default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:35.677733 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38b99ee757714ee0a3cfe0a22f476d81" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:35.679657 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 611d4317ec3249819fd98b7d59da87d6 (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
W20250626 19:58:35.680137 6703 catalog_manager.cc:4895] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 1abf7ec7ef164277bc0a23338a711027 because the tablet was not found. No further retry: Not found: Tablet not found: 1abf7ec7ef164277bc0a23338a711027
I20250626 19:58:35.680567 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 611d4317ec3249819fd98b7d59da87d6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:58:35.683429 7009 leader_election.cc:343] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.683825 7009 leader_election.cc:304] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:35.684793 7742 raft_consensus.cc:2747] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:35.690629 7742 raft_consensus.cc:491] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:35.691119 7742 raft_consensus.cc:513] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.692847 7742 leader_election.cc:290] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:35.696238 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edf78b9b864446f08056f25423942db3" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
W20250626 19:58:35.697520 7009 leader_election.cc:343] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.704633 7716 tablet_bootstrap.cc:492] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:35.705025 7716 ts_tablet_manager.cc:1397] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.235s user 0.008s sys 0.009s
I20250626 19:58:35.658603 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 49237fba2f414927b2e18124d5f9bece with delete_type TABLET_DATA_DELETED (replaced by tablet c91a8c1915eb40508b2c1482e03d6638) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:35.694343 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edf78b9b864446f08056f25423942db3" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:35.711323 7568 raft_consensus.cc:383] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.711621 7568 raft_consensus.cc:738] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.712273 7568 consensus_queue.cc:260] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:35.715711 6705 catalog_manager.cc:4908] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 1abf7ec7ef164277bc0a23338a711027 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1abf7ec7ef164277bc0a23338a711027 already in progress: creating tablet
W20250626 19:58:35.717665 6714 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 38b99ee757714ee0a3cfe0a22f476d81 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 38b99ee757714ee0a3cfe0a22f476d81 consensus state has no leader
I20250626 19:58:35.717965 7568 ts_tablet_manager.cc:1428] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.064s user 0.005s sys 0.000s
I20250626 19:58:35.703315 7560 tablet_replica.cc:331] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:35.705605 7733 raft_consensus.cc:491] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:35.718854 7716 raft_consensus.cc:357] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.719743 7716 raft_consensus.cc:383] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.719571 7733 raft_consensus.cc:513] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.720072 7716 raft_consensus.cc:738] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.721334 7733 leader_election.cc:290] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:35.721156 7716 consensus_queue.cc:260] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:35.721822 7215 raft_consensus.cc:2466] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:35.725329 7006 leader_election.cc:304] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7
I20250626 19:58:35.727735 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edf78b9b864446f08056f25423942db3" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:35.728288 7071 raft_consensus.cc:2466] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:35.729059 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edf78b9b864446f08056f25423942db3" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
W20250626 19:58:35.730441 7152 leader_election.cc:343] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.731393 7742 raft_consensus.cc:2802] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:35.731734 7742 raft_consensus.cc:491] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:35.732052 7742 raft_consensus.cc:3058] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:35.735253 7150 leader_election.cc:304] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7
I20250626 19:58:35.736289 7733 raft_consensus.cc:2802] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:35.736582 7733 raft_consensus.cc:491] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:35.736832 7733 raft_consensus.cc:3058] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:35.739104 6704 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 38b99ee757714ee0a3cfe0a22f476d81 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 38b99ee757714ee0a3cfe0a22f476d81 consensus state has no leader
I20250626 19:58:35.744292 7560 raft_consensus.cc:2241] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:35.744933 7560 pending_rounds.cc:70] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a: Trying to abort 1 pending ops.
I20250626 19:58:35.746995 7560 pending_rounds.cc:77] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964580077780992 op_type: NO_OP noop_request { }
I20250626 19:58:35.747316 7560 raft_consensus.cc:2887] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:35.747610 7560 raft_consensus.cc:2270] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:35.748055 7569 tablet_bootstrap.cc:492] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:35.748478 7569 ts_tablet_manager.cc:1397] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.199s user 0.022s sys 0.031s
I20250626 19:58:35.750022 7716 ts_tablet_manager.cc:1428] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20250626 19:58:35.750916 7569 raft_consensus.cc:357] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:35.751560 7569 raft_consensus.cc:383] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.751816 7569 raft_consensus.cc:738] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.752388 7569 consensus_queue.cc:260] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:35.756771 7453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 19:58:35.758967 7453 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 19:58:35.763078 7569 ts_tablet_manager.cc:1428] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.014s user 0.007s sys 0.001s
I20250626 19:58:35.765836 7569 tablet_bootstrap.cc:492] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:35.770738 7569 tablet_bootstrap.cc:654] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.764508 7568 tablet_bootstrap.cc:492] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:35.772109 7453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 19:58:35.786458 7569 tablet_bootstrap.cc:492] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:35.786897 7569 ts_tablet_manager.cc:1397] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.003s
I20250626 19:58:35.789175 7569 raft_consensus.cc:357] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:35.789753 7569 raft_consensus.cc:383] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.789994 7569 raft_consensus.cc:738] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.790808 7716 tablet_bootstrap.cc:492] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:35.791632 7742 raft_consensus.cc:513] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.793485 7742 leader_election.cc:290] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Requested vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:35.794309 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edf78b9b864446f08056f25423942db3" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:35.795495 7560 ts_tablet_manager.cc:1905] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:35.796100 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edf78b9b864446f08056f25423942db3" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
W20250626 19:58:35.797251 7009 leader_election.cc:343] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.800275 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 49237fba2f414927b2e18124d5f9bece with delete_type TABLET_DATA_DELETED (replaced by tablet c91a8c1915eb40508b2c1482e03d6638) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:35.801680 7569 consensus_queue.cc:260] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:35.802585 7473 tablet_replica.cc:331] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:35.803259 7473 raft_consensus.cc:2241] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:35.792717 7733 raft_consensus.cc:513] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.807233 7215 raft_consensus.cc:2391] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bbe151be01e24c82854d9c8c79c797ca in current term 1: Already voted for candidate 315d304854c541bf8b5e6b1eea712d4a in this term.
I20250626 19:58:35.808998 7006 leader_election.cc:304] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:35.810475 7569 ts_tablet_manager.cc:1428] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.023s user 0.004s sys 0.001s
I20250626 19:58:35.810813 7742 raft_consensus.cc:2747] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:58:35.811087 7568 tablet_bootstrap.cc:654] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.817683 7569 tablet_bootstrap.cc:492] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:35.824770 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edf78b9b864446f08056f25423942db3" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:35.825476 7071 raft_consensus.cc:2391] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 315d304854c541bf8b5e6b1eea712d4a in current term 1: Already voted for candidate bbe151be01e24c82854d9c8c79c797ca in this term.
I20250626 19:58:35.826964 7473 raft_consensus.cc:2270] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:35.828852 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edf78b9b864446f08056f25423942db3" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:35.829862 7569 tablet_bootstrap.cc:654] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.831414 7473 ts_tablet_manager.cc:1905] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:58:35.830160 7152 leader_election.cc:343] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.841557 7152 leader_election.cc:304] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a; no voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:35.842355 7655 raft_consensus.cc:2747] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:58:35.831771 7733 leader_election.cc:290] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:35.844328 7569 tablet_bootstrap.cc:492] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:35.844724 7569 ts_tablet_manager.cc:1397] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.007s
I20250626 19:58:35.847280 7716 tablet_bootstrap.cc:654] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.829639 7560 ts_tablet_manager.cc:1918] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:35.847847 7560 log.cc:1199] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/7a7b126a6af840adb0c42c5b137620a6
I20250626 19:58:35.847734 7569 raft_consensus.cc:357] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.848371 7569 raft_consensus.cc:383] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.848649 7569 raft_consensus.cc:738] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.849300 7569 consensus_queue.cc:260] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.854308 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 101f870043e44be7b3bf098c37b108af from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:35.857936 7560 ts_tablet_manager.cc:1939] T 7a7b126a6af840adb0c42c5b137620a6 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:35.858306 7569 ts_tablet_manager.cc:1428] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.013s user 0.003s sys 0.003s
I20250626 19:58:35.858407 7568 tablet_bootstrap.cc:492] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:35.859042 7568 ts_tablet_manager.cc:1397] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.104s user 0.010s sys 0.000s
I20250626 19:58:35.860332 7560 tablet_replica.cc:331] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:35.861060 7560 raft_consensus.cc:2241] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:35.861516 7560 raft_consensus.cc:2270] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:35.861492 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 7a7b126a6af840adb0c42c5b137620a6 (table default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:35.864660 7560 ts_tablet_manager.cc:1905] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:35.866500 7742 raft_consensus.cc:491] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:35.867012 7742 raft_consensus.cc:513] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:35.869136 7742 leader_election.cc:290] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:35.870986 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "101f870043e44be7b3bf098c37b108af" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:35.872002 6714 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 38b99ee757714ee0a3cfe0a22f476d81 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
W20250626 19:58:35.873134 7006 leader_election.cc:343] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.875582 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "101f870043e44be7b3bf098c37b108af" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
W20250626 19:58:35.880200 7009 leader_election.cc:343] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:35.879971 7568 raft_consensus.cc:357] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.880595 7568 raft_consensus.cc:383] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.880615 7009 leader_election.cc:304] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:35.880851 7568 raft_consensus.cc:738] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.881626 7742 raft_consensus.cc:2747] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:35.881495 7568 consensus_queue.cc:260] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.889000 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 101f870043e44be7b3bf098c37b108af
I20250626 19:58:35.892445 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 101f870043e44be7b3bf098c37b108af with delete_type TABLET_DATA_DELETED (replaced by tablet 578cf59de39b427d8a3074d30e3aeb84) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:35.893107 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 101f870043e44be7b3bf098c37b108af deleted and replaced by tablet 578cf59de39b427d8a3074d30e3aeb84
I20250626 19:58:35.893177 7559 tablet_replica.cc:331] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:35.893889 7559 raft_consensus.cc:2241] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:35.894340 7559 raft_consensus.cc:2270] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:35.896814 7559 ts_tablet_manager.cc:1905] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:35.899463 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 101f870043e44be7b3bf098c37b108af with delete_type TABLET_DATA_DELETED (replaced by tablet 578cf59de39b427d8a3074d30e3aeb84) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:35.900089 7341 tablet_service.cc:1515] Processing DeleteTablet for tablet 101f870043e44be7b3bf098c37b108af with delete_type TABLET_DATA_DELETED (replaced by tablet 578cf59de39b427d8a3074d30e3aeb84) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:35.905242 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 0c02e3d2b20b4577b00d131cf08643bd (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.906219 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c02e3d2b20b4577b00d131cf08643bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.907773 7568 ts_tablet_manager.cc:1428] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.048s user 0.004s sys 0.001s
I20250626 19:58:35.908927 7568 tablet_bootstrap.cc:492] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:35.912955 7716 tablet_bootstrap.cc:492] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:35.913589 7716 ts_tablet_manager.cc:1397] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.123s user 0.012s sys 0.000s
I20250626 19:58:35.919003 7341 tablet_service.cc:1468] Processing CreateTablet for tablet 0c02e3d2b20b4577b00d131cf08643bd (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.920113 7341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c02e3d2b20b4577b00d131cf08643bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.923944 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 0c02e3d2b20b4577b00d131cf08643bd (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.925027 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c02e3d2b20b4577b00d131cf08643bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.928730 7473 ts_tablet_manager.cc:1918] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:35.929057 7473 log.cc:1199] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/49237fba2f414927b2e18124d5f9bece
I20250626 19:58:35.929631 7473 ts_tablet_manager.cc:1939] T 49237fba2f414927b2e18124d5f9bece P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:35.932134 6709 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 38b99ee757714ee0a3cfe0a22f476d81 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
I20250626 19:58:35.939015 7716 raft_consensus.cc:357] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.940002 7716 raft_consensus.cc:383] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:35.940428 7716 raft_consensus.cc:738] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:35.941414 7716 consensus_queue.cc:260] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:35.944528 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 734d9d3331b44fc49e1f64b8caf1e4d8 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:35.947404 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 49237fba2f414927b2e18124d5f9bece (table default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]) successfully deleted
W20250626 19:58:35.949685 6705 catalog_manager.cc:4908] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 101f870043e44be7b3bf098c37b108af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 101f870043e44be7b3bf098c37b108af already in progress: creating tablet
I20250626 19:58:35.966996 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 578cf59de39b427d8a3074d30e3aeb84 (DEFAULT_TABLE table=default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.968330 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578cf59de39b427d8a3074d30e3aeb84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.971880 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 578cf59de39b427d8a3074d30e3aeb84 (DEFAULT_TABLE table=default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:35.973125 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578cf59de39b427d8a3074d30e3aeb84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:35.978956 7569 tablet_bootstrap.cc:492] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:35.984357 7569 tablet_bootstrap.cc:654] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.996491 7569 tablet_bootstrap.cc:492] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:35.996429 7559 ts_tablet_manager.cc:1918] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:35.997044 7569 ts_tablet_manager.cc:1397] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.009s
I20250626 19:58:35.997265 7559 log.cc:1199] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/101f870043e44be7b3bf098c37b108af
I20250626 19:58:35.997884 7559 ts_tablet_manager.cc:1939] T 101f870043e44be7b3bf098c37b108af P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:35.999429 7569 raft_consensus.cc:357] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.000064 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 101f870043e44be7b3bf098c37b108af (table default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a]) successfully deleted
I20250626 19:58:36.000216 7569 raft_consensus.cc:383] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.000473 7569 raft_consensus.cc:738] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.001122 7569 consensus_queue.cc:260] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.002322 7568 tablet_bootstrap.cc:654] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:35.968818 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 578cf59de39b427d8a3074d30e3aeb84 (DEFAULT_TABLE table=default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:36.005332 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578cf59de39b427d8a3074d30e3aeb84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:36.006034 7569 ts_tablet_manager.cc:1428] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250626 19:58:36.008386 7716 ts_tablet_manager.cc:1428] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.094s user 0.006s sys 0.000s
I20250626 19:58:36.009275 7716 tablet_bootstrap.cc:492] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:36.014608 7733 raft_consensus.cc:491] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:36.015080 7733 raft_consensus.cc:513] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.016796 7733 leader_election.cc:290] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:36.017264 6714 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 0c02e3d2b20b4577b00d131cf08643bd (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:36.018378 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73d091b5175d49398beb3ea231a93f63" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
W20250626 19:58:36.018689 6714 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 0c02e3d2b20b4577b00d131cf08643bd (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 0c02e3d2b20b4577b00d131cf08643bd consensus state has no leader
I20250626 19:58:36.019042 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73d091b5175d49398beb3ea231a93f63" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:36.019625 7071 raft_consensus.cc:2466] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
W20250626 19:58:36.019784 7152 leader_election.cc:343] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:36.020750 7150 leader_election.cc:304] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7
I20250626 19:58:36.022150 7733 raft_consensus.cc:2802] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:36.022517 7733 raft_consensus.cc:491] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:36.022861 7733 raft_consensus.cc:3058] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:36.029475 6714 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 0c02e3d2b20b4577b00d131cf08643bd (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:36.035243 7569 tablet_bootstrap.cc:492] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:36.036562 7733 raft_consensus.cc:513] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.040814 7569 tablet_bootstrap.cc:654] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.041136 7560 ts_tablet_manager.cc:1918] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:36.041496 7560 log.cc:1199] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/1abf7ec7ef164277bc0a23338a711027
I20250626 19:58:36.042114 7560 ts_tablet_manager.cc:1939] T 1abf7ec7ef164277bc0a23338a711027 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:36.043895 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73d091b5175d49398beb3ea231a93f63" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:36.044327 7341 tablet_service.cc:1515] Processing DeleteTablet for tablet 1abf7ec7ef164277bc0a23338a711027 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c02e3d2b20b4577b00d131cf08643bd) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:36.044451 7071 raft_consensus.cc:3058] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:36.044965 7473 tablet_replica.cc:331] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:36.045722 7473 raft_consensus.cc:2241] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:36.046218 7473 raft_consensus.cc:2270] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.049237 7473 ts_tablet_manager.cc:1905] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.054068 7071 raft_consensus.cc:2466] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:36.056011 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 1abf7ec7ef164277bc0a23338a711027 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
W20250626 19:58:36.056739 6702 catalog_manager.cc:4895] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet 49237fba2f414927b2e18124d5f9bece because the tablet was not found. No further retry: Not found: Tablet not found: 49237fba2f414927b2e18124d5f9bece
W20250626 19:58:36.057451 6702 catalog_manager.cc:4908] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet 101f870043e44be7b3bf098c37b108af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 101f870043e44be7b3bf098c37b108af already in progress: creating tablet
I20250626 19:58:36.067150 7150 leader_election.cc:304] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:36.067955 7655 raft_consensus.cc:2802] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:36.080564 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73d091b5175d49398beb3ea231a93f63" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
W20250626 19:58:36.082118 7152 leader_election.cc:343] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:36.084663 7568 tablet_bootstrap.cc:492] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:36.085197 7568 ts_tablet_manager.cc:1397] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.176s user 0.010s sys 0.000s
I20250626 19:58:36.087396 7568 raft_consensus.cc:357] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.088019 7568 raft_consensus.cc:383] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.088805 7568 raft_consensus.cc:738] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.052670 7716 tablet_bootstrap.cc:654] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.089376 7733 leader_election.cc:290] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:36.092016 7568 consensus_queue.cc:260] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.105647 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 611d4317ec3249819fd98b7d59da87d6 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:36.110069 7655 raft_consensus.cc:695] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:36.111291 7655 consensus_queue.cc:237] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.132548 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 611d4317ec3249819fd98b7d59da87d6
I20250626 19:58:36.134518 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 611d4317ec3249819fd98b7d59da87d6 with delete_type TABLET_DATA_DELETED (replaced by tablet e267bd75d7d342a3a6c28b17eb37d83e) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:36.135002 7341 tablet_service.cc:1515] Processing DeleteTablet for tablet 611d4317ec3249819fd98b7d59da87d6 with delete_type TABLET_DATA_DELETED (replaced by tablet e267bd75d7d342a3a6c28b17eb37d83e) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:36.135694 7559 tablet_replica.cc:331] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:36.135700 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 611d4317ec3249819fd98b7d59da87d6 deleted and replaced by tablet e267bd75d7d342a3a6c28b17eb37d83e
I20250626 19:58:36.136365 7559 raft_consensus.cc:2241] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:36.136770 7559 raft_consensus.cc:2270] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.137385 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 611d4317ec3249819fd98b7d59da87d6 with delete_type TABLET_DATA_DELETED (replaced by tablet e267bd75d7d342a3a6c28b17eb37d83e) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:36.138818 7559 ts_tablet_manager.cc:1905] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:58:36.139075 6702 catalog_manager.cc:4908] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet 611d4317ec3249819fd98b7d59da87d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 611d4317ec3249819fd98b7d59da87d6 already in progress: creating tablet
I20250626 19:58:36.105675 7568 ts_tablet_manager.cc:1428] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20250626 19:58:36.153138 7569 tablet_bootstrap.cc:492] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:36.153491 7569 ts_tablet_manager.cc:1397] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.118s user 0.012s sys 0.009s
I20250626 19:58:36.153776 7568 tablet_bootstrap.cc:492] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:36.155795 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 9ee6604fe87a43398a9545ea70ce19c4 from {username='slave'} at 127.0.0.1:42822
W20250626 19:58:36.158512 6704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250626 19:58:36.161302 6704 consensus_peers.cc:487] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f -> Peer 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417): Couldn't send request to peer 8ae38ec946e14198ae4adc8dde9d66e8. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250626 19:58:36.164501 7473 ts_tablet_manager.cc:1918] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:36.164871 7716 tablet_bootstrap.cc:492] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:36.165084 7692 raft_consensus.cc:491] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:36.165344 7716 ts_tablet_manager.cc:1397] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.156s user 0.012s sys 0.000s
I20250626 19:58:36.165526 7692 raft_consensus.cc:513] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:36.166041 7569 raft_consensus.cc:357] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.166632 7569 raft_consensus.cc:383] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.166882 7569 raft_consensus.cc:738] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.167321 7692 leader_election.cc:290] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:36.167420 7569 consensus_queue.cc:260] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.168818 7716 raft_consensus.cc:357] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.169416 7716 raft_consensus.cc:383] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.169413 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17591b5d114f474283965d67d0339153" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:36.169656 7716 raft_consensus.cc:738] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.172902 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17591b5d114f474283965d67d0339153" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:36.173607 7215 raft_consensus.cc:2391] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 297d2c913667408bae82668a557e06c7 in current term 1: Already voted for candidate bbe151be01e24c82854d9c8c79c797ca in this term.
I20250626 19:58:36.176206 7293 leader_election.cc:304] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 297d2c913667408bae82668a557e06c7; no voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:36.177271 7692 raft_consensus.cc:3058] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:36.181694 7559 ts_tablet_manager.cc:1918] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:36.182029 7559 log.cc:1199] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/611d4317ec3249819fd98b7d59da87d6
I20250626 19:58:36.182597 7559 ts_tablet_manager.cc:1939] T 611d4317ec3249819fd98b7d59da87d6 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:36.182442 7716 consensus_queue.cc:260] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.185385 7569 ts_tablet_manager.cc:1428] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20250626 19:58:36.189881 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 611d4317ec3249819fd98b7d59da87d6 (table default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:36.193429 7692 raft_consensus.cc:2747] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:36.206887 7051 tablet_service.cc:1468] Processing CreateTablet for tablet e267bd75d7d342a3a6c28b17eb37d83e (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:36.207835 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e267bd75d7d342a3a6c28b17eb37d83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:36.207734 7196 tablet_service.cc:1468] Processing CreateTablet for tablet e267bd75d7d342a3a6c28b17eb37d83e (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:36.208653 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e267bd75d7d342a3a6c28b17eb37d83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:36.211886 7568 tablet_bootstrap.cc:654] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.221064 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 9ee6604fe87a43398a9545ea70ce19c4
I20250626 19:58:36.216846 6716 catalog_manager.cc:3460] renamed table default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49] in HMS: new name default.table_4e311e30e4244e30aa0506381ffc07c0
I20250626 19:58:36.222865 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ee6604fe87a43398a9545ea70ce19c4 with delete_type TABLET_DATA_DELETED (replaced by tablet e95e6754e74a491087c4ea48e06c1e97) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:36.223513 7560 tablet_replica.cc:331] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:36.224045 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 9ee6604fe87a43398a9545ea70ce19c4 deleted and replaced by tablet e95e6754e74a491087c4ea48e06c1e97
I20250626 19:58:36.224192 7560 raft_consensus.cc:2241] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:36.224619 7560 raft_consensus.cc:2270] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.226032 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ee6604fe87a43398a9545ea70ce19c4 with delete_type TABLET_DATA_DELETED (replaced by tablet e95e6754e74a491087c4ea48e06c1e97) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:36.226701 7559 tablet_replica.cc:331] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:36.227394 7559 raft_consensus.cc:2241] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:36.227787 7559 raft_consensus.cc:2270] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.228085 7560 ts_tablet_manager.cc:1905] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.229418 7559 ts_tablet_manager.cc:1905] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.239048 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ee6604fe87a43398a9545ea70ce19c4 with delete_type TABLET_DATA_DELETED (replaced by tablet e95e6754e74a491087c4ea48e06c1e97) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:36.242921 7569 tablet_bootstrap.cc:492] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:36.248461 7569 tablet_bootstrap.cc:654] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.250878 7339 tablet_service.cc:1468] Processing CreateTablet for tablet e267bd75d7d342a3a6c28b17eb37d83e (DEFAULT_TABLE table=default.table_f9f594403d34451f8f2fa6d607d33a2c [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:36.252511 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e267bd75d7d342a3a6c28b17eb37d83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:36.258625 6714 catalog_manager.cc:5582] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130). New cstate: current_term: 1 leader_uuid: "315d304854c541bf8b5e6b1eea712d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:36.261317 7569 tablet_bootstrap.cc:492] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:36.261734 7569 ts_tablet_manager.cc:1397] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.000s
I20250626 19:58:36.262642 7473 log.cc:1199] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/1abf7ec7ef164277bc0a23338a711027
I20250626 19:58:36.264089 7569 raft_consensus.cc:357] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.264710 7569 raft_consensus.cc:383] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.264951 7569 raft_consensus.cc:738] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.265848 7569 consensus_queue.cc:260] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.263358 7473 ts_tablet_manager.cc:1939] T 1abf7ec7ef164277bc0a23338a711027 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:36.267547 7716 ts_tablet_manager.cc:1428] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.102s user 0.008s sys 0.000s
I20250626 19:58:36.268627 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 1abf7ec7ef164277bc0a23338a711027 (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:36.267464 7781 raft_consensus.cc:491] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:36.269186 7781 raft_consensus.cc:513] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.271102 7781 leader_election.cc:290] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
W20250626 19:58:36.271617 6717 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a43e8ff4d1844c0b9dddf28fc310fd8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:36.274085 7569 ts_tablet_manager.cc:1428] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.012s user 0.004s sys 0.002s
W20250626 19:58:36.274068 6711 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ced1a26248c748fb8c06cd348d12fb19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:58:36.275817 6713 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6bf87f28d4d34102bf1d02e2e1465a33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:36.278770 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edf78b9b864446f08056f25423942db3" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:36.279485 7071 raft_consensus.cc:2391] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 297d2c913667408bae82668a557e06c7 in current term 1: Already voted for candidate bbe151be01e24c82854d9c8c79c797ca in this term.
I20250626 19:58:36.281168 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edf78b9b864446f08056f25423942db3" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:36.281880 7215 raft_consensus.cc:2391] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 297d2c913667408bae82668a557e06c7 in current term 1: Already voted for candidate 315d304854c541bf8b5e6b1eea712d4a in this term.
I20250626 19:58:36.285776 7293 leader_election.cc:304] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 297d2c913667408bae82668a557e06c7; no voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:36.289326 7716 tablet_bootstrap.cc:492] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:36.292279 7781 raft_consensus.cc:3058] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:36.326232 6705 catalog_manager.cc:4908] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 611d4317ec3249819fd98b7d59da87d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 611d4317ec3249819fd98b7d59da87d6 already in progress: creating tablet
I20250626 19:58:36.327003 7781 raft_consensus.cc:2747] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:58:36.328666 6705 catalog_manager.cc:4908] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 9ee6604fe87a43398a9545ea70ce19c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ee6604fe87a43398a9545ea70ce19c4 already in progress: creating tablet
I20250626 19:58:36.329071 7559 ts_tablet_manager.cc:1918] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:36.334336 7742 consensus_queue.cc:1035] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:36.335958 7742 consensus_queue.cc:1035] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:36.348047 6714 master_service.cc:948] ReplaceTablet: received request to replace tablet 734d9d3331b44fc49e1f64b8caf1e4d8 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:36.360282 7560 ts_tablet_manager.cc:1918] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:36.361974 7716 tablet_bootstrap.cc:654] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.362866 7559 log.cc:1199] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/9ee6604fe87a43398a9545ea70ce19c4
I20250626 19:58:36.363600 7559 ts_tablet_manager.cc:1939] T 9ee6604fe87a43398a9545ea70ce19c4 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:36.365993 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 9ee6604fe87a43398a9545ea70ce19c4 (table default.table_76cdf24ffacc4d129b66c2fd6c0b93d8 [id=ad857025d66f4893872c8b5c92a9c0c7]) successfully deleted
I20250626 19:58:36.371186 7341 tablet_service.cc:1468] Processing CreateTablet for tablet e95e6754e74a491087c4ea48e06c1e97 (DEFAULT_TABLE table=default.table_76cdf24ffacc4d129b66c2fd6c0b93d8 [id=ad857025d66f4893872c8b5c92a9c0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:36.372402 7341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e95e6754e74a491087c4ea48e06c1e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:36.383121 7716 tablet_bootstrap.cc:492] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:36.383565 7716 ts_tablet_manager.cc:1397] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.106s user 0.010s sys 0.000s
W20250626 19:58:36.391752 6944 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e267bd75d7d342a3a6c28b17eb37d83e consensus state has no leader
W20250626 19:58:36.393610 6712 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fa425acbc3844433a95ac1c8ee1dc26d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:36.398610 6716 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42822 (request call id 64) took 2787 ms. Trace:
I20250626 19:58:36.402892 7560 log.cc:1199] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/9ee6604fe87a43398a9545ea70ce19c4
I20250626 19:58:36.402417 6716 rpcz_store.cc:276] 0626 19:58:33.611367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:33.696543 (+ 85176us) service_pool.cc:225] Handling call
0626 19:58:35.321794 (+1625251us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:36.395814 (+1074020us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:36.397749 (+ 1935us) catalog_manager.cc:3683] Apply alter schema
0626 19:58:36.398550 (+ 801us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":18146,"Hive Metastore.run_cpu_time_us":1781,"Hive Metastore.run_wall_time_us":875685}
I20250626 19:58:36.403517 7560 ts_tablet_manager.cc:1939] T 9ee6604fe87a43398a9545ea70ce19c4 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:36.405814 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 9ee6604fe87a43398a9545ea70ce19c4 (table default.table_76cdf24ffacc4d129b66c2fd6c0b93d8 [id=ad857025d66f4893872c8b5c92a9c0c7]) successfully deleted
I20250626 19:58:36.386035 7716 raft_consensus.cc:357] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.411806 7716 raft_consensus.cc:383] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.412084 7716 raft_consensus.cc:738] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.412719 7716 consensus_queue.cc:260] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.415768 7051 tablet_service.cc:1468] Processing CreateTablet for tablet e95e6754e74a491087c4ea48e06c1e97 (DEFAULT_TABLE table=default.table_76cdf24ffacc4d129b66c2fd6c0b93d8 [id=ad857025d66f4893872c8b5c92a9c0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:36.416757 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e95e6754e74a491087c4ea48e06c1e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:36.418215 7196 tablet_service.cc:1468] Processing CreateTablet for tablet e95e6754e74a491087c4ea48e06c1e97 (DEFAULT_TABLE table=default.table_76cdf24ffacc4d129b66c2fd6c0b93d8 [id=ad857025d66f4893872c8b5c92a9c0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:36.419214 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e95e6754e74a491087c4ea48e06c1e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:36.435137 6716 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 38b99ee757714ee0a3cfe0a22f476d81 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
I20250626 19:58:36.443797 7716 ts_tablet_manager.cc:1428] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.060s user 0.007s sys 0.000s
I20250626 19:58:36.449542 7716 tablet_bootstrap.cc:492] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:36.450238 7569 tablet_bootstrap.cc:492] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:36.450278 6714 master_service.cc:948] ReplaceTablet: received request to replace tablet 17591b5d114f474283965d67d0339153 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:36.455693 7569 tablet_bootstrap.cc:654] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.460160 6709 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 38b99ee757714ee0a3cfe0a22f476d81 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
I20250626 19:58:36.465216 7569 tablet_bootstrap.cc:492] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:36.465639 7569 ts_tablet_manager.cc:1397] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.001s
I20250626 19:58:36.466768 7716 tablet_bootstrap.cc:654] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.468171 7569 raft_consensus.cc:357] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:36.470850 7569 raft_consensus.cc:383] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.471151 7569 raft_consensus.cc:738] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.472754 6714 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 17591b5d114f474283965d67d0339153
I20250626 19:58:36.473534 7569 consensus_queue.cc:260] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:36.474195 7341 tablet_service.cc:1515] Processing DeleteTablet for tablet 17591b5d114f474283965d67d0339153 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ce27ec457e246d8bb4f693934932810) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:36.475250 7473 tablet_replica.cc:331] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:36.475852 7473 raft_consensus.cc:2241] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:36.476305 7473 raft_consensus.cc:2270] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.475997 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 17591b5d114f474283965d67d0339153 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ce27ec457e246d8bb4f693934932810) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:36.476502 6714 catalog_manager.cc:6652] ReplaceTablet: tablet 17591b5d114f474283965d67d0339153 deleted and replaced by tablet 8ce27ec457e246d8bb4f693934932810
I20250626 19:58:36.476838 7559 tablet_replica.cc:331] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:36.477403 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 17591b5d114f474283965d67d0339153 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ce27ec457e246d8bb4f693934932810) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:36.477511 7559 raft_consensus.cc:2241] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:36.478261 7559 raft_consensus.cc:2270] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.480702 7559 ts_tablet_manager.cc:1905] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.484864 7716 tablet_bootstrap.cc:492] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:36.485284 7716 ts_tablet_manager.cc:1397] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.003s
I20250626 19:58:36.487696 7716 raft_consensus.cc:357] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.488282 7716 raft_consensus.cc:383] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.488533 7716 raft_consensus.cc:738] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.492136 7569 ts_tablet_manager.cc:1428] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.026s user 0.004s sys 0.000s
I20250626 19:58:36.500638 7716 consensus_queue.cc:260] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.511174 7716 ts_tablet_manager.cc:1428] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20250626 19:58:36.515268 7716 tablet_bootstrap.cc:492] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:36.519925 7559 ts_tablet_manager.cc:1918] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:36.520291 7559 log.cc:1199] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/17591b5d114f474283965d67d0339153
I20250626 19:58:36.521003 7559 ts_tablet_manager.cc:1939] T 17591b5d114f474283965d67d0339153 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:36.521878 7473 ts_tablet_manager.cc:1905] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.523262 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 17591b5d114f474283965d67d0339153 (table default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]) successfully deleted
I20250626 19:58:36.529098 7655 raft_consensus.cc:491] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:36.529531 7655 raft_consensus.cc:513] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.531731 7341 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ee6604fe87a43398a9545ea70ce19c4 with delete_type TABLET_DATA_DELETED (replaced by tablet e95e6754e74a491087c4ea48e06c1e97) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:36.531275 7655 leader_election.cc:290] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:36.533983 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38b99ee757714ee0a3cfe0a22f476d81" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:36.534551 7071 raft_consensus.cc:2466] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:36.535370 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 8ce27ec457e246d8bb4f693934932810 (DEFAULT_TABLE table=default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:36.535670 7150 leader_election.cc:304] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:36.536326 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce27ec457e246d8bb4f693934932810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:36.536504 7655 raft_consensus.cc:2802] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:36.536779 7655 raft_consensus.cc:491] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:36.537039 7655 raft_consensus.cc:3058] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:36.539516 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38b99ee757714ee0a3cfe0a22f476d81" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:36.540127 7360 raft_consensus.cc:2466] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:36.541733 7560 tablet_replica.cc:331] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:36.542675 7560 raft_consensus.cc:2241] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:36.546329 7716 tablet_bootstrap.cc:654] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.543227 7655 raft_consensus.cc:513] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.546850 7560 raft_consensus.cc:2270] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.544446 7568 tablet_bootstrap.cc:492] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:36.547437 7568 ts_tablet_manager.cc:1397] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.394s user 0.008s sys 0.005s
I20250626 19:58:36.552889 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 8ce27ec457e246d8bb4f693934932810 (DEFAULT_TABLE table=default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:36.553176 7560 ts_tablet_manager.cc:1905] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.553795 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce27ec457e246d8bb4f693934932810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:36.559018 6714 master_service.cc:948] ReplaceTablet: received request to replace tablet e267bd75d7d342a3a6c28b17eb37d83e from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:36.564591 7655 leader_election.cc:290] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:36.565460 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38b99ee757714ee0a3cfe0a22f476d81" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:36.564780 7568 raft_consensus.cc:357] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.565755 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38b99ee757714ee0a3cfe0a22f476d81" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:36.566053 7568 raft_consensus.cc:383] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.566004 7071 raft_consensus.cc:3058] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:36.566319 7568 raft_consensus.cc:738] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.566959 7360 raft_consensus.cc:3058] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:36.567538 7568 consensus_queue.cc:260] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.569556 7568 ts_tablet_manager.cc:1428] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.022s user 0.003s sys 0.001s
I20250626 19:58:36.584112 7568 tablet_bootstrap.cc:492] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
W20250626 19:58:36.586289 6716 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
I20250626 19:58:36.588165 7742 raft_consensus.cc:491] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:36.588634 7742 raft_consensus.cc:513] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.590458 7742 leader_election.cc:290] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:36.591925 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c91a8c1915eb40508b2c1482e03d6638" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:36.592343 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c91a8c1915eb40508b2c1482e03d6638" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
W20250626 19:58:36.593581 7009 leader_election.cc:343] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:36.596143 6714 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet e267bd75d7d342a3a6c28b17eb37d83e
W20250626 19:58:36.597420 7006 leader_election.cc:343] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:36.597618 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet e267bd75d7d342a3a6c28b17eb37d83e with delete_type TABLET_DATA_DELETED (replaced by tablet 17d9cd26171943088ad676722edf9577) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:36.597808 7006 leader_election.cc:304] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:36.598380 7559 tablet_replica.cc:331] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:36.599120 7559 raft_consensus.cc:2241] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:36.599382 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet e267bd75d7d342a3a6c28b17eb37d83e with delete_type TABLET_DATA_DELETED (replaced by tablet 17d9cd26171943088ad676722edf9577) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:36.599542 7559 raft_consensus.cc:2270] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.600041 7742 raft_consensus.cc:2747] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:36.600229 6714 catalog_manager.cc:6652] ReplaceTablet: tablet e267bd75d7d342a3a6c28b17eb37d83e deleted and replaced by tablet 17d9cd26171943088ad676722edf9577
I20250626 19:58:36.601082 7341 tablet_service.cc:1515] Processing DeleteTablet for tablet e267bd75d7d342a3a6c28b17eb37d83e with delete_type TABLET_DATA_DELETED (replaced by tablet 17d9cd26171943088ad676722edf9577) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:36.601675 7360 raft_consensus.cc:2466] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:36.601949 7559 ts_tablet_manager.cc:1905] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.602846 7152 leader_election.cc:304] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:36.603639 7655 raft_consensus.cc:2802] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:36.605969 6714 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_76cdf24ffacc4d129b66c2fd6c0b93d8" } new_table_name: "default.Table_d04e336d056d484ab051754a2b012d9b" modify_external_catalogs: true
I20250626 19:58:36.607314 7655 raft_consensus.cc:695] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:36.608062 7655 consensus_queue.cc:237] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.615595 7568 tablet_bootstrap.cc:654] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.616567 7071 raft_consensus.cc:2466] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:36.619591 7716 tablet_bootstrap.cc:492] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:36.620004 7716 ts_tablet_manager.cc:1397] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.105s user 0.009s sys 0.003s
I20250626 19:58:36.622123 7716 raft_consensus.cc:357] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.625540 7341 tablet_service.cc:1468] Processing CreateTablet for tablet 8ce27ec457e246d8bb4f693934932810 (DEFAULT_TABLE table=default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:36.626709 7341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ce27ec457e246d8bb4f693934932810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:36.629487 7473 ts_tablet_manager.cc:1918] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:36.629853 7473 log.cc:1199] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/17591b5d114f474283965d67d0339153
I20250626 19:58:36.630492 7473 ts_tablet_manager.cc:1939] T 17591b5d114f474283965d67d0339153 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:36.632431 7568 tablet_bootstrap.cc:492] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:36.632797 7568 ts_tablet_manager.cc:1397] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.001s
I20250626 19:58:36.633700 7559 ts_tablet_manager.cc:1918] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:36.634073 7559 log.cc:1199] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/e267bd75d7d342a3a6c28b17eb37d83e
I20250626 19:58:36.636915 6716 catalog_manager.cc:5582] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130). New cstate: current_term: 1 leader_uuid: "315d304854c541bf8b5e6b1eea712d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:36.638549 6716 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 38b99ee757714ee0a3cfe0a22f476d81 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
I20250626 19:58:36.622714 7716 raft_consensus.cc:383] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.634707 7559 ts_tablet_manager.cc:1939] T e267bd75d7d342a3a6c28b17eb37d83e P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:36.643719 7716 raft_consensus.cc:738] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.645010 7716 consensus_queue.cc:260] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.645792 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet e267bd75d7d342a3a6c28b17eb37d83e (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:36.651552 7568 raft_consensus.cc:357] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.660058 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 17591b5d114f474283965d67d0339153 (table default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]) successfully deleted
I20250626 19:58:36.666090 7716 ts_tablet_manager.cc:1428] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.046s user 0.006s sys 0.000s
I20250626 19:58:36.666968 7568 raft_consensus.cc:383] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.667344 7473 tablet_replica.cc:331] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:36.667907 7568 raft_consensus.cc:738] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.668022 7473 raft_consensus.cc:2241] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:36.668437 7473 raft_consensus.cc:2270] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.673426 7716 tablet_bootstrap.cc:492] T 101f870043e44be7b3bf098c37b108af P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:36.674535 7568 consensus_queue.cc:260] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.677925 7742 raft_consensus.cc:491] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:36.678376 7742 raft_consensus.cc:513] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.680245 7742 leader_election.cc:290] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:36.681685 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c02e3d2b20b4577b00d131cf08643bd" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
W20250626 19:58:36.682973 7009 leader_election.cc:343] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:36.684640 7568 ts_tablet_manager.cc:1428] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.052s user 0.003s sys 0.003s
I20250626 19:58:36.693571 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 73d091b5175d49398beb3ea231a93f63 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:36.687831 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c02e3d2b20b4577b00d131cf08643bd" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
W20250626 19:58:36.695175 7006 leader_election.cc:343] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:36.695632 7006 leader_election.cc:304] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:36.696162 7569 tablet_bootstrap.cc:492] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:36.665014 7560 ts_tablet_manager.cc:1918] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:36.696379 7742 raft_consensus.cc:2747] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:36.696853 7560 log.cc:1199] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/17591b5d114f474283965d67d0339153
I20250626 19:58:36.697664 7560 ts_tablet_manager.cc:1939] T 17591b5d114f474283965d67d0339153 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:36.700109 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 17591b5d114f474283965d67d0339153 (table default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]) successfully deleted
W20250626 19:58:36.701701 6702 catalog_manager.cc:4908] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet e267bd75d7d342a3a6c28b17eb37d83e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e267bd75d7d342a3a6c28b17eb37d83e already in progress: creating tablet
I20250626 19:58:36.701898 7569 tablet_bootstrap.cc:654] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.704218 7568 tablet_bootstrap.cc:492] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:36.715147 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 73d091b5175d49398beb3ea231a93f63
I20250626 19:58:36.721869 7569 tablet_bootstrap.cc:492] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:36.722363 7569 ts_tablet_manager.cc:1397] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.012s
I20250626 19:58:36.725458 7716 tablet_bootstrap.cc:654] T 101f870043e44be7b3bf098c37b108af P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.725095 7569 raft_consensus.cc:357] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.725729 7569 raft_consensus.cc:383] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.725971 7569 raft_consensus.cc:738] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.726693 7569 consensus_queue.cc:260] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.729974 7569 ts_tablet_manager.cc:1428] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250626 19:58:36.733788 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 73d091b5175d49398beb3ea231a93f63 with delete_type TABLET_DATA_DELETED (replaced by tablet 7bf0c724d4684773b40aff24524e1e92) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:36.739022 7559 tablet_replica.cc:331] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:36.739938 7559 raft_consensus.cc:2241] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:36.740455 7559 raft_consensus.cc:2270] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.742129 7473 ts_tablet_manager.cc:1905] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.742218 7559 ts_tablet_manager.cc:1905] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.748163 7568 tablet_bootstrap.cc:654] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.750265 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 73d091b5175d49398beb3ea231a93f63 with delete_type TABLET_DATA_DELETED (replaced by tablet 7bf0c724d4684773b40aff24524e1e92) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:36.750921 7560 tablet_replica.cc:331] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:36.751540 7560 raft_consensus.cc:2241] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:36.752172 7560 pending_rounds.cc:70] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a: Trying to abort 1 pending ops.
I20250626 19:58:36.752370 7560 pending_rounds.cc:77] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964584524218368 op_type: NO_OP noop_request { }
I20250626 19:58:36.752660 7560 raft_consensus.cc:2887] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:36.752964 7560 raft_consensus.cc:2270] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.753285 7742 raft_consensus.cc:491] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:36.753671 7742 raft_consensus.cc:513] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.755380 7742 leader_election.cc:290] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 19:58:36.756013 6704 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e267bd75d7d342a3a6c28b17eb37d83e (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e267bd75d7d342a3a6c28b17eb37d83e consensus state has no leader
I20250626 19:58:36.757071 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578cf59de39b427d8a3074d30e3aeb84" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:36.757351 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578cf59de39b427d8a3074d30e3aeb84" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
W20250626 19:58:36.758512 7009 leader_election.cc:343] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:58:36.759059 7006 leader_election.cc:343] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:36.759406 7006 leader_election.cc:304] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:36.759241 7560 ts_tablet_manager.cc:1905] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.760293 7742 raft_consensus.cc:2747] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:36.762375 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 73d091b5175d49398beb3ea231a93f63 deleted and replaced by tablet 7bf0c724d4684773b40aff24524e1e92
I20250626 19:58:36.765470 7341 tablet_service.cc:1515] Processing DeleteTablet for tablet 73d091b5175d49398beb3ea231a93f63 with delete_type TABLET_DATA_DELETED (replaced by tablet 7bf0c724d4684773b40aff24524e1e92) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:36.771817 7716 tablet_bootstrap.cc:492] T 101f870043e44be7b3bf098c37b108af P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:36.772367 7716 ts_tablet_manager.cc:1397] T 101f870043e44be7b3bf098c37b108af P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.099s user 0.009s sys 0.004s
I20250626 19:58:36.774647 7716 raft_consensus.cc:357] T 101f870043e44be7b3bf098c37b108af P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.775314 7716 raft_consensus.cc:383] T 101f870043e44be7b3bf098c37b108af P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.775554 7716 raft_consensus.cc:738] T 101f870043e44be7b3bf098c37b108af P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.776054 7473 ts_tablet_manager.cc:1918] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:36.776398 7473 log.cc:1199] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/9ee6604fe87a43398a9545ea70ce19c4
I20250626 19:58:36.776266 7716 consensus_queue.cc:260] T 101f870043e44be7b3bf098c37b108af P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.776995 7473 ts_tablet_manager.cc:1939] T 9ee6604fe87a43398a9545ea70ce19c4 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:36.779323 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 9ee6604fe87a43398a9545ea70ce19c4 (table default.table_76cdf24ffacc4d129b66c2fd6c0b93d8 [id=ad857025d66f4893872c8b5c92a9c0c7]) successfully deleted
W20250626 19:58:36.780901 6705 catalog_manager.cc:4908] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet e267bd75d7d342a3a6c28b17eb37d83e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e267bd75d7d342a3a6c28b17eb37d83e already in progress: creating tablet
I20250626 19:58:36.781934 7473 tablet_replica.cc:331] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:36.788832 7473 raft_consensus.cc:2241] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:36.789302 7473 raft_consensus.cc:2270] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.792407 7473 ts_tablet_manager.cc:1905] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.802116 7716 ts_tablet_manager.cc:1428] T 101f870043e44be7b3bf098c37b108af P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20250626 19:58:36.803005 7716 tablet_bootstrap.cc:492] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
W20250626 19:58:36.805401 6716 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
I20250626 19:58:36.808183 7716 tablet_bootstrap.cc:654] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.812415 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 17d9cd26171943088ad676722edf9577 (DEFAULT_TABLE table=default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:36.813390 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d9cd26171943088ad676722edf9577. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:36.816118 7473 ts_tablet_manager.cc:1918] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:36.816474 7473 log.cc:1199] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/73d091b5175d49398beb3ea231a93f63
I20250626 19:58:36.817031 7473 ts_tablet_manager.cc:1939] T 73d091b5175d49398beb3ea231a93f63 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:36.818222 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 38b99ee757714ee0a3cfe0a22f476d81 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:36.819877 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 73d091b5175d49398beb3ea231a93f63 (table default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]) successfully deleted
I20250626 19:58:36.821599 7341 tablet_service.cc:1468] Processing CreateTablet for tablet 17d9cd26171943088ad676722edf9577 (DEFAULT_TABLE table=default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:36.828994 7569 tablet_bootstrap.cc:492] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:36.834272 7569 tablet_bootstrap.cc:654] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.838681 6715 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]
I20250626 19:58:36.842026 7568 tablet_bootstrap.cc:492] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:36.842444 7568 ts_tablet_manager.cc:1397] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.138s user 0.012s sys 0.002s
I20250626 19:58:36.843772 7716 tablet_bootstrap.cc:492] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:36.844336 7716 ts_tablet_manager.cc:1397] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.001s
W20250626 19:58:36.847086 6716 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
I20250626 19:58:36.847288 7716 raft_consensus.cc:357] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.847882 7716 raft_consensus.cc:383] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.848126 7716 raft_consensus.cc:738] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.853320 7341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d9cd26171943088ad676722edf9577. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:36.853966 7568 raft_consensus.cc:357] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.854576 7568 raft_consensus.cc:383] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.854843 7568 raft_consensus.cc:738] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.857378 7568 consensus_queue.cc:260] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.859681 7560 ts_tablet_manager.cc:1918] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:36.860700 7716 consensus_queue.cc:260] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:36.863919 7716 ts_tablet_manager.cc:1428] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20250626 19:58:36.872606 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 38b99ee757714ee0a3cfe0a22f476d81
I20250626 19:58:36.873836 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 38b99ee757714ee0a3cfe0a22f476d81 deleted and replaced by tablet 60a00cf0b1854a20b54c4ae8a59ede29
I20250626 19:58:36.874143 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 38b99ee757714ee0a3cfe0a22f476d81 with delete_type TABLET_DATA_DELETED (replaced by tablet 60a00cf0b1854a20b54c4ae8a59ede29) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:36.874918 7716 tablet_bootstrap.cc:492] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:36.875130 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 38b99ee757714ee0a3cfe0a22f476d81 with delete_type TABLET_DATA_DELETED (replaced by tablet 60a00cf0b1854a20b54c4ae8a59ede29) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:36.875433 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 17d9cd26171943088ad676722edf9577 (DEFAULT_TABLE table=default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:36.876278 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d9cd26171943088ad676722edf9577. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:36.881471 7716 tablet_bootstrap.cc:654] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.884440 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 38b99ee757714ee0a3cfe0a22f476d81 with delete_type TABLET_DATA_DELETED (replaced by tablet 60a00cf0b1854a20b54c4ae8a59ede29) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:36.886541 7473 tablet_replica.cc:331] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:36.887221 7559 ts_tablet_manager.cc:1918] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:36.887377 7473 raft_consensus.cc:2241] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:36.887578 7559 log.cc:1199] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/73d091b5175d49398beb3ea231a93f63
I20250626 19:58:36.887853 7473 raft_consensus.cc:2270] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.888197 7559 ts_tablet_manager.cc:1939] T 73d091b5175d49398beb3ea231a93f63 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:36.890331 7473 ts_tablet_manager.cc:1905] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.891091 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 73d091b5175d49398beb3ea231a93f63 (table default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]) successfully deleted
I20250626 19:58:36.899158 7559 tablet_replica.cc:331] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:36.899430 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 611d4317ec3249819fd98b7d59da87d6 with delete_type TABLET_DATA_DELETED (replaced by tablet e267bd75d7d342a3a6c28b17eb37d83e) from {username='slave'} at 127.0.0.1:48028
W20250626 19:58:36.901264 7152 consensus_peers.cc:487] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250626 19:58:36.901677 7150 consensus_peers.cc:487] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:58:36.903031 7568 ts_tablet_manager.cc:1428] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.060s user 0.007s sys 0.000s
I20250626 19:58:36.860025 7560 log.cc:1199] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/73d091b5175d49398beb3ea231a93f63
I20250626 19:58:36.909096 7560 ts_tablet_manager.cc:1939] T 73d091b5175d49398beb3ea231a93f63 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:36.909974 7569 tablet_bootstrap.cc:492] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:36.910372 7569 ts_tablet_manager.cc:1397] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.082s user 0.024s sys 0.039s
I20250626 19:58:36.913251 7569 raft_consensus.cc:357] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.913877 7569 raft_consensus.cc:383] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.914140 7569 raft_consensus.cc:738] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.916030 7473 ts_tablet_manager.cc:1918] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:36.917701 7473 log.cc:1199] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/38b99ee757714ee0a3cfe0a22f476d81
I20250626 19:58:36.918390 7473 ts_tablet_manager.cc:1939] T 38b99ee757714ee0a3cfe0a22f476d81 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:36.919494 7716 tablet_bootstrap.cc:492] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:36.921680 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 38b99ee757714ee0a3cfe0a22f476d81 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:36.922528 7716 ts_tablet_manager.cc:1397] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.003s
I20250626 19:58:36.924847 7568 tablet_bootstrap.cc:492] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:36.924870 7716 raft_consensus.cc:357] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.925855 7716 raft_consensus.cc:383] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:36.926170 7716 raft_consensus.cc:738] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:36.926959 7716 consensus_queue.cc:260] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.927340 7559 raft_consensus.cc:2241] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:36.927783 7559 raft_consensus.cc:2270] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.932243 7559 ts_tablet_manager.cc:1905] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.933804 7716 ts_tablet_manager.cc:1428] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.011s user 0.002s sys 0.002s
I20250626 19:58:36.939051 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 73d091b5175d49398beb3ea231a93f63 (table default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]) successfully deleted
I20250626 19:58:36.940125 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 611d4317ec3249819fd98b7d59da87d6 with delete_type TABLET_DATA_DELETED (replaced by tablet e267bd75d7d342a3a6c28b17eb37d83e) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:36.941546 7473 tablet_replica.cc:331] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:36.942238 7473 raft_consensus.cc:2241] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:36.942713 7473 raft_consensus.cc:2270] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.946362 7473 ts_tablet_manager.cc:1905] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.948328 7560 tablet_replica.cc:331] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:36.949107 7560 raft_consensus.cc:2241] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:36.954236 7716 tablet_bootstrap.cc:492] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:36.955058 7560 pending_rounds.cc:70] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a: Trying to abort 2 pending ops.
I20250626 19:58:36.955915 7569 consensus_queue.cc:260] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:36.955924 7560 pending_rounds.cc:77] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7171964587602964480 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "38b99ee757714ee0a3cfe0a22f476d81" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.table_252448e828024438ba317b8277b66523" dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" new_extra_config { } }
I20250626 19:58:36.957967 7569 ts_tablet_manager.cc:1428] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.047s user 0.005s sys 0.000s
W20250626 19:58:36.957952 7560 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48028: Aborted: Op aborted
W20250626 19:58:36.959818 6702 catalog_manager.cc:5007] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): alter failed for tablet 38b99ee757714ee0a3cfe0a22f476d81 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]): Aborted: Op aborted
I20250626 19:58:36.963325 7473 ts_tablet_manager.cc:1918] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:36.963666 7473 log.cc:1199] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/611d4317ec3249819fd98b7d59da87d6
I20250626 19:58:36.964299 7473 ts_tablet_manager.cc:1939] T 611d4317ec3249819fd98b7d59da87d6 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:36.930677 7568 tablet_bootstrap.cc:654] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.964155 7560 pending_rounds.cc:77] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964586436972544 op_type: NO_OP noop_request { }
I20250626 19:58:36.966665 7560 raft_consensus.cc:2887] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:36.967422 7560 raft_consensus.cc:2270] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:36.970096 7560 ts_tablet_manager.cc:1905] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:36.970302 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 611d4317ec3249819fd98b7d59da87d6 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:36.978228 6716 master_service.cc:948] ReplaceTablet: received request to replace tablet 0c02e3d2b20b4577b00d131cf08643bd from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:36.981438 7716 tablet_bootstrap.cc:654] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:36.991925 7568 tablet_bootstrap.cc:492] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:36.994796 7568 ts_tablet_manager.cc:1397] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.070s user 0.011s sys 0.000s
I20250626 19:58:36.999856 7559 ts_tablet_manager.cc:1918] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:37.001663 7568 raft_consensus.cc:357] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.009039 6716 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 0c02e3d2b20b4577b00d131cf08643bd
I20250626 19:58:37.010582 7559 log.cc:1199] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/38b99ee757714ee0a3cfe0a22f476d81
I20250626 19:58:37.011262 7559 ts_tablet_manager.cc:1939] T 38b99ee757714ee0a3cfe0a22f476d81 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:37.011485 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c02e3d2b20b4577b00d131cf08643bd with delete_type TABLET_DATA_DELETED (replaced by tablet 53819554bad8423fbaf39567acadfeda) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:37.011538 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c02e3d2b20b4577b00d131cf08643bd with delete_type TABLET_DATA_DELETED (replaced by tablet 53819554bad8423fbaf39567acadfeda) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:37.012223 6716 catalog_manager.cc:6652] ReplaceTablet: tablet 0c02e3d2b20b4577b00d131cf08643bd deleted and replaced by tablet 53819554bad8423fbaf39567acadfeda
I20250626 19:58:37.012336 7568 raft_consensus.cc:383] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.012984 7568 raft_consensus.cc:738] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.013530 7559 tablet_replica.cc:331] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:37.014171 7559 raft_consensus.cc:2241] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:37.014142 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 38b99ee757714ee0a3cfe0a22f476d81 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:37.014598 7559 raft_consensus.cc:2270] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:37.015134 7568 consensus_queue.cc:260] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.015846 7341 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c02e3d2b20b4577b00d131cf08643bd with delete_type TABLET_DATA_DELETED (replaced by tablet 53819554bad8423fbaf39567acadfeda) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:37.018127 7716 tablet_bootstrap.cc:492] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:37.018544 7716 ts_tablet_manager.cc:1397] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.064s user 0.013s sys 0.000s
W20250626 19:58:37.021163 6705 catalog_manager.cc:4908] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 0c02e3d2b20b4577b00d131cf08643bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c02e3d2b20b4577b00d131cf08643bd already in progress: creating tablet
I20250626 19:58:37.021255 7716 raft_consensus.cc:357] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.021879 7716 raft_consensus.cc:383] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.022116 7716 raft_consensus.cc:738] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.022758 7716 consensus_queue.cc:260] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.023835 6709 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 17d9cd26171943088ad676722edf9577 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
I20250626 19:58:37.025904 7742 raft_consensus.cc:491] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:37.026355 7742 raft_consensus.cc:513] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.028266 7742 leader_election.cc:290] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:37.029561 7716 ts_tablet_manager.cc:1428] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20250626 19:58:37.029670 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17d9cd26171943088ad676722edf9577" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:37.030368 7716 tablet_bootstrap.cc:492] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
W20250626 19:58:37.031044 7006 leader_election.cc:343] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:37.033452 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17d9cd26171943088ad676722edf9577" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:37.033737 7568 ts_tablet_manager.cc:1428] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
W20250626 19:58:37.033713 6709 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 17d9cd26171943088ad676722edf9577 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 17d9cd26171943088ad676722edf9577 consensus state has no leader
W20250626 19:58:37.034816 7009 leader_election.cc:343] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:37.035218 7009 leader_election.cc:304] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:37.036229 7742 raft_consensus.cc:2747] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:37.036834 7559 ts_tablet_manager.cc:1905] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:37.039211 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 7bf0c724d4684773b40aff24524e1e92 (DEFAULT_TABLE table=default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.040159 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bf0c724d4684773b40aff24524e1e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.043291 6709 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 17d9cd26171943088ad676722edf9577 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
I20250626 19:58:37.044216 7716 tablet_bootstrap.cc:654] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.058418 7341 tablet_service.cc:1468] Processing CreateTablet for tablet 7bf0c724d4684773b40aff24524e1e92 (DEFAULT_TABLE table=default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.058655 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 7bf0c724d4684773b40aff24524e1e92 (DEFAULT_TABLE table=default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.059397 7341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bf0c724d4684773b40aff24524e1e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.059525 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bf0c724d4684773b40aff24524e1e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.066689 7716 tablet_bootstrap.cc:492] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:37.067143 7716 ts_tablet_manager.cc:1397] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.001s
I20250626 19:58:37.069442 7716 raft_consensus.cc:357] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:37.070060 7716 raft_consensus.cc:383] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.070324 7716 raft_consensus.cc:738] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.072414 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 60a00cf0b1854a20b54c4ae8a59ede29 (DEFAULT_TABLE table=default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.073356 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60a00cf0b1854a20b54c4ae8a59ede29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.073885 7559 ts_tablet_manager.cc:1918] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:37.074205 7559 log.cc:1199] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/0c02e3d2b20b4577b00d131cf08643bd
I20250626 19:58:37.074790 7559 ts_tablet_manager.cc:1939] T 0c02e3d2b20b4577b00d131cf08643bd P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:37.075863 7568 tablet_bootstrap.cc:492] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:37.071075 7716 consensus_queue.cc:260] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:37.085050 7716 ts_tablet_manager.cc:1428] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.018s user 0.004s sys 0.001s
I20250626 19:58:37.085445 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 60a00cf0b1854a20b54c4ae8a59ede29 (DEFAULT_TABLE table=default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.085880 7716 tablet_bootstrap.cc:492] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:37.086547 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60a00cf0b1854a20b54c4ae8a59ede29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.092046 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 60a00cf0b1854a20b54c4ae8a59ede29 (DEFAULT_TABLE table=default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.093031 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60a00cf0b1854a20b54c4ae8a59ede29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.092945 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 0c02e3d2b20b4577b00d131cf08643bd (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:37.099095 7716 tablet_bootstrap.cc:654] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.107360 7340 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c02e3d2b20b4577b00d131cf08643bd with delete_type TABLET_DATA_DELETED (replaced by tablet 53819554bad8423fbaf39567acadfeda) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:37.108106 7473 tablet_replica.cc:331] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:37.108741 7473 raft_consensus.cc:2241] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:37.109133 7473 raft_consensus.cc:2270] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:37.112499 7473 ts_tablet_manager.cc:1905] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:37.115635 7655 raft_consensus.cc:491] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:37.116063 7655 raft_consensus.cc:513] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:37.117853 7655 leader_election.cc:290] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:37.119640 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "101f870043e44be7b3bf098c37b108af" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:37.120383 7360 raft_consensus.cc:2466] T 101f870043e44be7b3bf098c37b108af P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:37.121623 7152 leader_election.cc:304] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
W20250626 19:58:37.124559 7150 leader_election.cc:343] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af
I20250626 19:58:37.125653 7655 raft_consensus.cc:2802] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:37.126024 7655 raft_consensus.cc:491] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:37.126297 7655 raft_consensus.cc:3058] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:37.133682 7655 raft_consensus.cc:513] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:37.135134 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 8ce27ec457e246d8bb4f693934932810 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:37.137255 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "101f870043e44be7b3bf098c37b108af" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:37.137871 7360 raft_consensus.cc:3058] T 101f870043e44be7b3bf098c37b108af P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:37.144359 7360 raft_consensus.cc:2466] T 101f870043e44be7b3bf098c37b108af P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:37.145701 7152 leader_election.cc:304] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:37.146682 7776 raft_consensus.cc:2802] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:37.148633 7569 tablet_bootstrap.cc:492] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:37.154282 7569 tablet_bootstrap.cc:654] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:37.161533 7150 leader_election.cc:343] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af
I20250626 19:58:37.164408 7776 raft_consensus.cc:695] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:37.165349 7776 consensus_queue.cc:237] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:37.169512 7655 leader_election.cc:290] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:37.175347 7569 tablet_bootstrap.cc:492] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:37.175761 7569 ts_tablet_manager.cc:1397] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.000s
I20250626 19:58:37.177971 7569 raft_consensus.cc:357] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.187917 7473 ts_tablet_manager.cc:1918] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:37.188354 7473 log.cc:1199] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/0c02e3d2b20b4577b00d131cf08643bd
I20250626 19:58:37.188922 7473 ts_tablet_manager.cc:1939] T 0c02e3d2b20b4577b00d131cf08643bd P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:37.196259 7716 tablet_bootstrap.cc:492] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:37.196661 7716 ts_tablet_manager.cc:1397] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.111s user 0.024s sys 0.015s
I20250626 19:58:37.198191 7568 tablet_bootstrap.cc:654] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.199045 7716 raft_consensus.cc:357] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:37.199389 6716 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
I20250626 19:58:37.178567 7569 raft_consensus.cc:383] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.199626 7716 raft_consensus.cc:383] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.199745 7569 raft_consensus.cc:738] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.199877 7716 raft_consensus.cc:738] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.205235 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 8ce27ec457e246d8bb4f693934932810
I20250626 19:58:37.206825 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ce27ec457e246d8bb4f693934932810 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d815600b288406587e86d2643fa7a69) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:37.207515 7559 tablet_replica.cc:331] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:37.205899 7560 ts_tablet_manager.cc:1918] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:58:37.207918 7560 log.cc:1199] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/38b99ee757714ee0a3cfe0a22f476d81
I20250626 19:58:37.208168 7559 raft_consensus.cc:2241] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:37.208554 7559 raft_consensus.cc:2270] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:37.210355 7559 ts_tablet_manager.cc:1905] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:37.210306 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ce27ec457e246d8bb4f693934932810 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d815600b288406587e86d2643fa7a69) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:37.210853 7569 consensus_queue.cc:260] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.211504 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ce27ec457e246d8bb4f693934932810 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d815600b288406587e86d2643fa7a69) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:37.211793 7560 ts_tablet_manager.cc:1939] T 38b99ee757714ee0a3cfe0a22f476d81 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:37.218034 7569 ts_tablet_manager.cc:1428] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20250626 19:58:37.218837 7569 tablet_bootstrap.cc:492] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:37.219053 7560 tablet_replica.cc:331] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:37.219687 7560 raft_consensus.cc:2241] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:37.219723 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 38b99ee757714ee0a3cfe0a22f476d81 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:37.220088 7560 raft_consensus.cc:2270] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:37.224488 7560 ts_tablet_manager.cc:1905] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:37.228279 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 0c02e3d2b20b4577b00d131cf08643bd (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
W20250626 19:58:37.228974 6705 catalog_manager.cc:4908] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 8ce27ec457e246d8bb4f693934932810 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ce27ec457e246d8bb4f693934932810 already in progress: creating tablet
I20250626 19:58:37.227248 7716 consensus_queue.cc:260] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.239992 7716 ts_tablet_manager.cc:1428] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.043s user 0.005s sys 0.000s
I20250626 19:58:37.244486 7568 tablet_bootstrap.cc:492] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:37.244902 7568 ts_tablet_manager.cc:1397] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.169s user 0.007s sys 0.002s
I20250626 19:58:37.247114 7568 raft_consensus.cc:357] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:37.247681 7568 raft_consensus.cc:383] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.247925 7568 raft_consensus.cc:738] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.248615 7568 consensus_queue.cc:260] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:37.252918 7716 tablet_bootstrap.cc:492] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:37.263731 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 8ce27ec457e246d8bb4f693934932810 deleted and replaced by tablet 2d815600b288406587e86d2643fa7a69
I20250626 19:58:37.263726 7568 ts_tablet_manager.cc:1428] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250626 19:58:37.267668 7569 tablet_bootstrap.cc:654] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.268760 7716 tablet_bootstrap.cc:654] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.264963 7559 ts_tablet_manager.cc:1918] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:37.272056 7568 tablet_bootstrap.cc:492] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:37.272006 7559 log.cc:1199] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/8ce27ec457e246d8bb4f693934932810
W20250626 19:58:37.272745 6709 catalog_manager.cc:5415] Ignoring report from unknown tablet e267bd75d7d342a3a6c28b17eb37d83e
I20250626 19:58:37.272773 7559 ts_tablet_manager.cc:1939] T 8ce27ec457e246d8bb4f693934932810 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:37.275813 7742 raft_consensus.cc:491] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:37.276249 7742 raft_consensus.cc:513] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:37.277495 7568 tablet_bootstrap.cc:654] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.278052 7742 leader_election.cc:290] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:37.280088 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e95e6754e74a491087c4ea48e06c1e97" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:37.280092 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e95e6754e74a491087c4ea48e06c1e97" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:37.280916 7776 raft_consensus.cc:491] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 19:58:37.281297 7009 leader_election.cc:343] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:37.281342 7776 raft_consensus.cc:513] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:37.281919 7006 leader_election.cc:343] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:37.282299 7006 leader_election.cc:304] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:37.283195 7776 leader_election.cc:290] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:37.283289 7742 raft_consensus.cc:2747] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:37.284898 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 8ce27ec457e246d8bb4f693934932810 (table default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]) successfully deleted
W20250626 19:58:37.285800 7152 leader_election.cc:343] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: 0c02e3d2b20b4577b00d131cf08643bd
W20250626 19:58:37.286626 7150 leader_election.cc:343] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Not found: Tablet not found: 0c02e3d2b20b4577b00d131cf08643bd
I20250626 19:58:37.286998 7150 leader_election.cc:304] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a; no voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:37.287878 7776 raft_consensus.cc:2747] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:37.292223 6710 catalog_manager.cc:3460] renamed table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb] in HMS: new name default.table_daf65ec234534a069c2d1ad34f13952a
I20250626 19:58:37.305135 7569 tablet_bootstrap.cc:492] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:37.305541 7569 ts_tablet_manager.cc:1397] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.087s user 0.008s sys 0.000s
I20250626 19:58:37.307720 7569 raft_consensus.cc:357] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.308326 7569 raft_consensus.cc:383] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.308568 7569 raft_consensus.cc:738] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.310492 7716 tablet_bootstrap.cc:492] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:37.310927 7716 ts_tablet_manager.cc:1397] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.002s
I20250626 19:58:37.312356 7560 ts_tablet_manager.cc:1918] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:37.312779 7560 log.cc:1199] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/611d4317ec3249819fd98b7d59da87d6
I20250626 19:58:37.313453 7560 ts_tablet_manager.cc:1939] T 611d4317ec3249819fd98b7d59da87d6 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:37.313118 7716 raft_consensus.cc:357] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:37.313711 7716 raft_consensus.cc:383] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.313943 7716 raft_consensus.cc:738] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.314577 7716 consensus_queue.cc:260] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:37.322160 7716 ts_tablet_manager.cc:1428] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20250626 19:58:37.322530 7569 consensus_queue.cc:260] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.324514 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 611d4317ec3249819fd98b7d59da87d6 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:37.327827 7716 tablet_bootstrap.cc:492] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:37.331166 7560 tablet_replica.cc:331] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:37.331852 7560 raft_consensus.cc:2241] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:37.332283 7560 raft_consensus.cc:2270] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:37.334185 7560 ts_tablet_manager.cc:1905] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:37.336485 7568 tablet_bootstrap.cc:492] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:37.336503 7569 ts_tablet_manager.cc:1428] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20250626 19:58:37.336877 7568 ts_tablet_manager.cc:1397] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.065s user 0.009s sys 0.001s
I20250626 19:58:37.341765 7568 raft_consensus.cc:357] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.342386 7568 raft_consensus.cc:383] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.342624 7568 raft_consensus.cc:738] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.343636 7568 consensus_queue.cc:260] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:37.340938 6716 catalog_manager.cc:5415] Ignoring report from unknown tablet e267bd75d7d342a3a6c28b17eb37d83e
I20250626 19:58:37.349110 7776 raft_consensus.cc:491] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:37.349532 7776 raft_consensus.cc:513] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:37.351329 7776 leader_election.cc:290] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:37.353276 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c91a8c1915eb40508b2c1482e03d6638" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:37.353859 7071 raft_consensus.cc:2466] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:37.354856 7150 leader_election.cc:304] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:37.355618 7776 raft_consensus.cc:2802] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:37.355612 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c91a8c1915eb40508b2c1482e03d6638" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:37.355948 7776 raft_consensus.cc:491] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:37.356201 7360 raft_consensus.cc:2466] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:37.356424 7776 raft_consensus.cc:3058] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:37.357142 7716 tablet_bootstrap.cc:654] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.364593 7568 ts_tablet_manager.cc:1428] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250626 19:58:37.364476 7776 raft_consensus.cc:513] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:37.366972 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c91a8c1915eb40508b2c1482e03d6638" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:37.367523 7360 raft_consensus.cc:3058] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:37.369565 7568 tablet_bootstrap.cc:492] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:37.374696 7716 tablet_bootstrap.cc:492] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:37.375131 7716 ts_tablet_manager.cc:1397] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.004s
I20250626 19:58:37.375468 7360 raft_consensus.cc:2466] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:37.376473 7152 leader_election.cc:304] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:37.377259 7829 raft_consensus.cc:2802] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:37.377668 7829 raft_consensus.cc:695] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:37.377416 7716 raft_consensus.cc:357] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.378029 7716 raft_consensus.cc:383] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.378273 7716 raft_consensus.cc:738] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.383576 7716 consensus_queue.cc:260] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.384307 7781 raft_consensus.cc:491] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:37.384696 7781 raft_consensus.cc:513] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.378428 7829 consensus_queue.cc:237] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
W20250626 19:58:37.386477 6709 catalog_manager.cc:5415] Ignoring report from unknown tablet e267bd75d7d342a3a6c28b17eb37d83e
I20250626 19:58:37.386830 7781 leader_election.cc:290] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:37.388717 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ce27ec457e246d8bb4f693934932810" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:37.389830 6709 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 60a00cf0b1854a20b54c4ae8a59ede29 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
W20250626 19:58:37.391382 7293 leader_election.cc:343] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:58:37.391592 6709 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 60a00cf0b1854a20b54c4ae8a59ede29 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 60a00cf0b1854a20b54c4ae8a59ede29 consensus state has no leader
W20250626 19:58:37.392818 7294 leader_election.cc:343] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Not found: Tablet not found: 8ce27ec457e246d8bb4f693934932810
I20250626 19:58:37.393174 7294 leader_election.cc:304] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 297d2c913667408bae82668a557e06c7; no voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:37.393494 6709 master_service.cc:948] ReplaceTablet: received request to replace tablet 17d9cd26171943088ad676722edf9577 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:37.394367 7781 raft_consensus.cc:2747] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:37.398382 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 2d815600b288406587e86d2643fa7a69 (DEFAULT_TABLE table=default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.399362 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d815600b288406587e86d2643fa7a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.399638 7716 ts_tablet_manager.cc:1428] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20250626 19:58:37.412424 6713 catalog_manager.cc:2149] created HMS catalog entry for table [id=58d985607f30440ea82dab74e20a1f8f]
I20250626 19:58:37.416632 7716 tablet_bootstrap.cc:492] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:37.417497 7568 tablet_bootstrap.cc:654] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.427083 7716 tablet_bootstrap.cc:654] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.427515 6713 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 75) took 2531 ms. Trace:
I20250626 19:58:37.427832 6713 rpcz_store.cc:276] 0626 19:58:34.895892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:34.896663 (+ 771us) service_pool.cc:225] Handling call
0626 19:58:36.276300 (+1379637us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:36.292514 (+ 16214us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:37.412387 (+1119873us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:37.426880 (+ 14493us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:37.427169 (+ 289us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:37.427438 (+ 269us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:37.414211 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:37.414574 (+ 363us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:37.414596 (+ 22us) write_op.cc:435] Acquired schema lock
0626 19:58:37.414607 (+ 11us) tablet.cc:660] Decoding operations
0626 19:58:37.414903 (+ 296us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:37.414944 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:37.414967 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:37.415180 (+ 213us) tablet.cc:699] Row locks acquired
0626 19:58:37.415192 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:58:37.415313 (+ 121us) write_op.cc:270] Start()
0626 19:58:37.415390 (+ 77us) write_op.cc:276] Timestamp: P: 1750967917415346 usec, L: 0
0626 19:58:37.415404 (+ 14us) op_driver.cc:348] REPLICATION: starting
0626 19:58:37.415880 (+ 476us) log.cc:844] Serialized 433 byte log entry
0626 19:58:37.424795 (+ 8915us) op_driver.cc:464] REPLICATION: finished
0626 19:58:37.425230 (+ 435us) write_op.cc:301] APPLY: starting
0626 19:58:37.425284 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:37.425380 (+ 96us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:37.425465 (+ 85us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:37.425837 (+ 372us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:37.425960 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:37.425998 (+ 38us) write_op.cc:312] APPLY: finished
0626 19:58:37.426156 (+ 158us) log.cc:844] Serialized 29 byte log entry
0626 19:58:37.426275 (+ 119us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:37.426338 (+ 63us) write_op.cc:454] Released schema lock
0626 19:58:37.426371 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0626 19:58:37.426564 (+ 193us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:37.426585 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":998744,"Hive Metastore.run_cpu_time_us":575,"Hive Metastore.run_wall_time_us":119994,"child_traces":[["op",{"apply.queue_time_us":289,"apply.run_cpu_time_us":1305,"apply.run_wall_time_us":1304,"num_ops":2,"prepare.queue_time_us":215,"prepare.run_cpu_time_us":2003,"prepare.run_wall_time_us":2645,"raft.queue_time_us":1237,"raft.run_cpu_time_us":538,"raft.run_wall_time_us":538,"replication_time_us":9314}]]}
W20250626 19:58:37.440498 6713 catalog_manager.cc:5415] Ignoring report from unknown tablet e267bd75d7d342a3a6c28b17eb37d83e
I20250626 19:58:37.441788 6713 catalog_manager.cc:5582] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130). New cstate: current_term: 1 leader_uuid: "315d304854c541bf8b5e6b1eea712d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:37.444885 7568 tablet_bootstrap.cc:492] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:37.445294 7568 ts_tablet_manager.cc:1397] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.000s
I20250626 19:58:37.447387 7568 raft_consensus.cc:357] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:37.447942 7568 raft_consensus.cc:383] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.448175 7568 raft_consensus.cc:738] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.448741 7568 consensus_queue.cc:260] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:37.452061 7716 tablet_bootstrap.cc:492] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:37.452436 7716 ts_tablet_manager.cc:1397] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.005s
I20250626 19:58:37.454581 7716 raft_consensus.cc:357] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.455225 7716 raft_consensus.cc:383] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.455461 7716 raft_consensus.cc:738] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.456249 7716 consensus_queue.cc:260] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.465878 7568 ts_tablet_manager.cc:1428] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250626 19:58:37.469218 7716 ts_tablet_manager.cc:1428] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.017s user 0.004s sys 0.002s
I20250626 19:58:37.478176 7716 tablet_bootstrap.cc:492] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:37.480168 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c91a8c1915eb40508b2c1482e03d6638" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:37.480710 7071 raft_consensus.cc:3058] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:37.481573 7568 tablet_bootstrap.cc:492] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:37.483615 7716 tablet_bootstrap.cc:654] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.487098 7560 ts_tablet_manager.cc:1918] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:37.487428 7560 log.cc:1199] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/0c02e3d2b20b4577b00d131cf08643bd
I20250626 19:58:37.487751 7071 raft_consensus.cc:2466] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:37.488024 7560 ts_tablet_manager.cc:1939] T 0c02e3d2b20b4577b00d131cf08643bd P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:37.489277 6711 catalog_manager.cc:2149] created HMS catalog entry for table [id=3f20c2a68db64ff7afbc98716ecf20b0]
I20250626 19:58:37.490522 7776 leader_election.cc:290] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:37.501272 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 0c02e3d2b20b4577b00d131cf08643bd (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
W20250626 19:58:37.503629 6702 catalog_manager.cc:4908] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet 8ce27ec457e246d8bb4f693934932810 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ce27ec457e246d8bb4f693934932810 already in progress: creating tablet
I20250626 19:58:37.506539 6711 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 73) took 2801 ms. Trace:
I20250626 19:58:37.507342 7781 raft_consensus.cc:491] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:37.507776 7781 raft_consensus.cc:513] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:37.509618 7781 leader_election.cc:290] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:37.506925 6711 rpcz_store.cc:276] 0626 19:58:34.704708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:34.708093 (+ 3385us) service_pool.cc:225] Handling call
0626 19:58:36.274456 (+1566363us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:36.293381 (+ 18925us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:37.489237 (+1195856us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:37.505878 (+ 16641us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:37.506199 (+ 321us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:37.506476 (+ 277us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:37.490891 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:37.491285 (+ 394us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:37.491306 (+ 21us) write_op.cc:435] Acquired schema lock
0626 19:58:37.491318 (+ 12us) tablet.cc:660] Decoding operations
0626 19:58:37.491569 (+ 251us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:37.491607 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:37.491628 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:37.491822 (+ 194us) tablet.cc:699] Row locks acquired
0626 19:58:37.491832 (+ 10us) write_op.cc:260] PREPARE: finished
0626 19:58:37.491933 (+ 101us) write_op.cc:270] Start()
0626 19:58:37.492009 (+ 76us) write_op.cc:276] Timestamp: P: 1750967917491985 usec, L: 0
0626 19:58:37.492024 (+ 15us) op_driver.cc:348] REPLICATION: starting
0626 19:58:37.494525 (+ 2501us) log.cc:844] Serialized 433 byte log entry
0626 19:58:37.502477 (+ 7952us) op_driver.cc:464] REPLICATION: finished
0626 19:58:37.504490 (+ 2013us) write_op.cc:301] APPLY: starting
0626 19:58:37.504544 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:37.504639 (+ 95us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:37.504729 (+ 90us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:37.505128 (+ 399us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:37.505269 (+ 141us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:37.505309 (+ 40us) write_op.cc:312] APPLY: finished
0626 19:58:37.505465 (+ 156us) log.cc:844] Serialized 29 byte log entry
0626 19:58:37.505592 (+ 127us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:37.505658 (+ 66us) write_op.cc:454] Released schema lock
0626 19:58:37.505692 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":1118061,"Hive Metastore.run_cpu_time_us":580,"Hive Metastore.run_wall_time_us":77285,"child_traces":[["op",{"apply.queue_time_us":1864,"num_ops":2,"prepare.queue_time_us":154,"prepare.run_cpu_time_us":1920,"prepare.run_wall_time_us":4596,"raft.queue_time_us":72,"raft.run_cpu_time_us":523,"raft.run_wall_time_us":523,"replication_time_us":10377}]]}
I20250626 19:58:37.511909 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578cf59de39b427d8a3074d30e3aeb84" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:37.512471 7071 raft_consensus.cc:2466] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:37.513479 7294 leader_election.cc:304] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:37.514247 7781 raft_consensus.cc:2802] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:37.514561 7781 raft_consensus.cc:491] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:37.514910 7781 raft_consensus.cc:3058] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:37.520752 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet c91a8c1915eb40508b2c1482e03d6638 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:37.521291 7781 raft_consensus.cc:513] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:37.528895 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578cf59de39b427d8a3074d30e3aeb84" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:37.529397 7071 raft_consensus.cc:3058] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:37.534852 7781 leader_election.cc:290] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:37.539168 7716 tablet_bootstrap.cc:492] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:37.539549 7716 ts_tablet_manager.cc:1397] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.000s
I20250626 19:58:37.541642 7716 raft_consensus.cc:357] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.542227 7716 raft_consensus.cc:383] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.542445 7716 raft_consensus.cc:738] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.543238 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet c91a8c1915eb40508b2c1482e03d6638
I20250626 19:58:37.542991 7716 consensus_queue.cc:260] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.545022 7716 ts_tablet_manager.cc:1428] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250626 19:58:37.546007 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet c91a8c1915eb40508b2c1482e03d6638 with delete_type TABLET_DATA_DELETED (replaced by tablet fabf0275e32e400c9731eb09d9c2620c) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:37.511101 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578cf59de39b427d8a3074d30e3aeb84" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:37.547088 7473 tablet_replica.cc:331] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:37.546913 7215 raft_consensus.cc:2466] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:37.486613 7568 tablet_bootstrap.cc:654] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.547729 7473 raft_consensus.cc:2241] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:37.548103 7473 raft_consensus.cc:2270] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:37.549088 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet c91a8c1915eb40508b2c1482e03d6638 with delete_type TABLET_DATA_DELETED (replaced by tablet fabf0275e32e400c9731eb09d9c2620c) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:37.549578 7473 ts_tablet_manager.cc:1905] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:37.523811 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578cf59de39b427d8a3074d30e3aeb84" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:37.550469 7216 raft_consensus.cc:3058] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:37.551354 7559 tablet_replica.cc:331] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:37.551442 6711 catalog_manager.cc:6652] ReplaceTablet: tablet c91a8c1915eb40508b2c1482e03d6638 deleted and replaced by tablet fabf0275e32e400c9731eb09d9c2620c
I20250626 19:58:37.551980 7559 raft_consensus.cc:2241] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:37.552352 7559 raft_consensus.cc:2270] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:37.553846 7559 ts_tablet_manager.cc:1905] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:37.556010 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 2d815600b288406587e86d2643fa7a69 (DEFAULT_TABLE table=default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.557053 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 2d815600b288406587e86d2643fa7a69 (DEFAULT_TABLE table=default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.544926 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet c91a8c1915eb40508b2c1482e03d6638 with delete_type TABLET_DATA_DELETED (replaced by tablet fabf0275e32e400c9731eb09d9c2620c) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:37.557999 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d815600b288406587e86d2643fa7a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.558135 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d815600b288406587e86d2643fa7a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.563544 7560 tablet_replica.cc:331] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:37.564262 7560 raft_consensus.cc:2241] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:37.564905 7560 pending_rounds.cc:70] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a: Trying to abort 1 pending ops.
I20250626 19:58:37.565120 7560 pending_rounds.cc:77] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964589615968256 op_type: NO_OP noop_request { }
I20250626 19:58:37.565433 7560 raft_consensus.cc:2887] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:37.565719 7560 raft_consensus.cc:2270] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:37.568150 7560 ts_tablet_manager.cc:1905] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:37.568993 7071 raft_consensus.cc:2466] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:37.570395 7294 leader_election.cc:304] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:37.572027 7781 raft_consensus.cc:2802] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:37.573976 7569 tablet_bootstrap.cc:492] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:37.574186 7216 raft_consensus.cc:2466] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:37.579308 7569 tablet_bootstrap.cc:654] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.581622 7716 tablet_bootstrap.cc:492] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:37.589716 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 53819554bad8423fbaf39567acadfeda (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.590692 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53819554bad8423fbaf39567acadfeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.593138 7559 ts_tablet_manager.cc:1918] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:37.593461 7559 log.cc:1199] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/c91a8c1915eb40508b2c1482e03d6638
I20250626 19:58:37.594034 7559 ts_tablet_manager.cc:1939] T c91a8c1915eb40508b2c1482e03d6638 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:37.607941 7569 tablet_bootstrap.cc:492] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:37.608994 7568 tablet_bootstrap.cc:492] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:37.609367 7568 ts_tablet_manager.cc:1397] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.128s user 0.014s sys 0.000s
I20250626 19:58:37.608350 7569 ts_tablet_manager.cc:1397] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.000s
I20250626 19:58:37.611680 7742 raft_consensus.cc:491] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:37.611598 7568 raft_consensus.cc:357] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.612174 7568 raft_consensus.cc:383] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.612286 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet c91a8c1915eb40508b2c1482e03d6638 (table default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]) successfully deleted
I20250626 19:58:37.612401 7568 raft_consensus.cc:738] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.612121 7742 raft_consensus.cc:513] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.612967 7568 consensus_queue.cc:260] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.613837 7742 leader_election.cc:290] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:37.614516 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7bf0c724d4684773b40aff24524e1e92" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:37.611680 7569 raft_consensus.cc:357] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.615276 7569 raft_consensus.cc:383] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.615514 7569 raft_consensus.cc:738] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
W20250626 19:58:37.616070 7006 leader_election.cc:343] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:37.616359 7569 consensus_queue.cc:260] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.607194 7781 raft_consensus.cc:695] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:37.618089 7781 consensus_queue.cc:237] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } }
I20250626 19:58:37.605289 7473 ts_tablet_manager.cc:1918] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:37.620994 7568 ts_tablet_manager.cc:1428] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250626 19:58:37.621256 7473 log.cc:1199] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/c91a8c1915eb40508b2c1482e03d6638
I20250626 19:58:37.616955 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7bf0c724d4684773b40aff24524e1e92" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
W20250626 19:58:37.623340 7009 leader_election.cc:343] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:37.623809 7009 leader_election.cc:304] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:37.624809 7716 tablet_bootstrap.cc:654] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.629953 7568 tablet_bootstrap.cc:492] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:37.636543 7560 ts_tablet_manager.cc:1918] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:37.636901 7560 log.cc:1199] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/c91a8c1915eb40508b2c1482e03d6638
I20250626 19:58:37.638123 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 53819554bad8423fbaf39567acadfeda (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.638887 7742 raft_consensus.cc:2747] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:37.639070 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53819554bad8423fbaf39567acadfeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.640692 7569 ts_tablet_manager.cc:1428] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.031s user 0.004s sys 0.000s
I20250626 19:58:37.646835 6713 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_6f04f2a6bf414cb8912fe28aff625d2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:37.621867 7473 ts_tablet_manager.cc:1939] T c91a8c1915eb40508b2c1482e03d6638 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:37.649907 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet c91a8c1915eb40508b2c1482e03d6638 (table default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]) successfully deleted
I20250626 19:58:37.655336 6711 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 60a00cf0b1854a20b54c4ae8a59ede29 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
I20250626 19:58:37.655459 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 53819554bad8423fbaf39567acadfeda (DEFAULT_TABLE table=default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.656553 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53819554bad8423fbaf39567acadfeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.657290 6717 catalog_manager.cc:2149] created HMS catalog entry for table [id=ef672ba275b54933ae739bf9e560ae68]
I20250626 19:58:37.667616 7341 tablet_service.cc:1515] Processing DeleteTablet for tablet 17d9cd26171943088ad676722edf9577 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b9a0f71535a4a65a775f728edbcc636) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:37.668256 7340 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ce27ec457e246d8bb4f693934932810 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d815600b288406587e86d2643fa7a69) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:37.669129 6709 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 17d9cd26171943088ad676722edf9577
I20250626 19:58:37.670960 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 17d9cd26171943088ad676722edf9577 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b9a0f71535a4a65a775f728edbcc636) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:37.671665 7559 tablet_replica.cc:331] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:37.673681 7716 tablet_bootstrap.cc:492] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:37.674324 7716 ts_tablet_manager.cc:1397] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.106s user 0.007s sys 0.004s
I20250626 19:58:37.674579 7340 tablet_service.cc:1515] Processing DeleteTablet for tablet 17d9cd26171943088ad676722edf9577 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b9a0f71535a4a65a775f728edbcc636) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:37.675508 6709 catalog_manager.cc:6652] ReplaceTablet: tablet 17d9cd26171943088ad676722edf9577 deleted and replaced by tablet 7b9a0f71535a4a65a775f728edbcc636
I20250626 19:58:37.676447 7716 raft_consensus.cc:357] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.677454 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ce27ec457e246d8bb4f693934932810 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d815600b288406587e86d2643fa7a69) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:37.678915 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 17d9cd26171943088ad676722edf9577 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b9a0f71535a4a65a775f728edbcc636) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:37.681939 7568 tablet_bootstrap.cc:654] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.677066 7716 raft_consensus.cc:383] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.687031 7716 raft_consensus.cc:738] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.687919 7716 consensus_queue.cc:260] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.682363 7560 ts_tablet_manager.cc:1939] T c91a8c1915eb40508b2c1482e03d6638 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:37.690527 6717 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 68) took 3424 ms. Trace:
I20250626 19:58:37.690948 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet c91a8c1915eb40508b2c1482e03d6638 (table default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]) successfully deleted
I20250626 19:58:37.691141 7560 tablet_replica.cc:331] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:37.691751 7560 raft_consensus.cc:2241] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:37.692143 7560 raft_consensus.cc:2270] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:37.690881 6717 rpcz_store.cc:276] 0626 19:58:34.265938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:34.373248 (+107310us) service_pool.cc:225] Handling call
0626 19:58:36.272035 (+1898787us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:36.315262 (+ 43227us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:37.657249 (+1341987us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:37.689811 (+ 32562us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:37.690180 (+ 369us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:37.690463 (+ 283us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:37.658922 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:37.659353 (+ 431us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:37.659374 (+ 21us) write_op.cc:435] Acquired schema lock
0626 19:58:37.659386 (+ 12us) tablet.cc:660] Decoding operations
0626 19:58:37.659633 (+ 247us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:37.659669 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:37.659691 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:37.659873 (+ 182us) tablet.cc:699] Row locks acquired
0626 19:58:37.659886 (+ 13us) write_op.cc:260] PREPARE: finished
0626 19:58:37.659989 (+ 103us) write_op.cc:270] Start()
0626 19:58:37.660064 (+ 75us) write_op.cc:276] Timestamp: P: 1750967917659971 usec, L: 0
0626 19:58:37.660087 (+ 23us) op_driver.cc:348] REPLICATION: starting
0626 19:58:37.660530 (+ 443us) log.cc:844] Serialized 433 byte log entry
0626 19:58:37.687902 (+ 27372us) op_driver.cc:464] REPLICATION: finished
0626 19:58:37.688319 (+ 417us) write_op.cc:301] APPLY: starting
0626 19:58:37.688386 (+ 67us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:37.688494 (+ 108us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:37.688578 (+ 84us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:37.688987 (+ 409us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:37.689124 (+ 137us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:37.689162 (+ 38us) write_op.cc:312] APPLY: finished
0626 19:58:37.689324 (+ 162us) log.cc:844] Serialized 29 byte log entry
0626 19:58:37.689475 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:37.689549 (+ 74us) write_op.cc:454] Released schema lock
0626 19:58:37.689578 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0626 19:58:37.689791 (+ 213us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:37.689814 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1173456,"Hive Metastore.run_cpu_time_us":551,"Hive Metastore.run_wall_time_us":167879,"child_traces":[["op",{"apply.queue_time_us":274,"apply.run_cpu_time_us":1441,"apply.run_wall_time_us":1440,"num_ops":2,"prepare.queue_time_us":169,"prepare.run_cpu_time_us":1880,"prepare.run_wall_time_us":1880,"raft.queue_time_us":147,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":12,"replication_time_us":27730}]]}
I20250626 19:58:37.698490 7559 raft_consensus.cc:2241] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:37.698987 7559 raft_consensus.cc:2270] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:37.669130 6711 catalog_manager.cc:5582] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:37.700623 7559 ts_tablet_manager.cc:1905] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:37.700928 7473 tablet_replica.cc:331] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:37.701649 7560 ts_tablet_manager.cc:1905] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:37.701941 7716 ts_tablet_manager.cc:1428] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20250626 19:58:37.702476 7569 tablet_bootstrap.cc:492] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:37.707720 7569 tablet_bootstrap.cc:654] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.707365 7716 tablet_bootstrap.cc:492] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:37.714187 6717 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_63a5ae538eef4f739a22697dfcfb56c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:37.714840 7568 tablet_bootstrap.cc:492] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:37.715260 7568 ts_tablet_manager.cc:1397] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.085s user 0.009s sys 0.001s
I20250626 19:58:37.715591 7473 raft_consensus.cc:2241] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:37.716113 7473 raft_consensus.cc:2270] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:37.721415 7716 tablet_bootstrap.cc:654] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.721979 7742 raft_consensus.cc:491] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:37.722419 7742 raft_consensus.cc:513] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.724319 7742 leader_election.cc:290] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:37.728332 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60a00cf0b1854a20b54c4ae8a59ede29" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
W20250626 19:58:37.729609 7006 leader_election.cc:343] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:37.731258 7569 tablet_bootstrap.cc:492] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:37.731655 7569 ts_tablet_manager.cc:1397] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.000s
I20250626 19:58:37.736296 7568 raft_consensus.cc:357] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.736971 7568 raft_consensus.cc:383] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.737222 7568 raft_consensus.cc:738] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.737819 7568 consensus_queue.cc:260] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.724076 7473 ts_tablet_manager.cc:1905] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:37.727685 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60a00cf0b1854a20b54c4ae8a59ede29" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:37.738782 7360 raft_consensus.cc:2466] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:37.740077 7009 leader_election.cc:304] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters: 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:37.741142 7742 raft_consensus.cc:2802] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:37.741449 7742 raft_consensus.cc:491] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:37.741730 7742 raft_consensus.cc:3058] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:37.734508 7569 raft_consensus.cc:357] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.746070 7569 raft_consensus.cc:383] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.746352 7569 raft_consensus.cc:738] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.747004 7569 consensus_queue.cc:260] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.756742 7568 ts_tablet_manager.cc:1428] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20250626 19:58:37.756395 7742 raft_consensus.cc:513] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.758188 7568 tablet_bootstrap.cc:492] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:37.758540 7559 ts_tablet_manager.cc:1918] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:37.759228 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60a00cf0b1854a20b54c4ae8a59ede29" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:37.759706 7559 log.cc:1199] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/17d9cd26171943088ad676722edf9577
I20250626 19:58:37.760352 7559 ts_tablet_manager.cc:1939] T 17d9cd26171943088ad676722edf9577 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:37.762459 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60a00cf0b1854a20b54c4ae8a59ede29" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:37.763024 7360 raft_consensus.cc:3058] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:37.762526 7006 leader_election.cc:343] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:37.769220 7360 raft_consensus.cc:2466] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:37.771162 7742 leader_election.cc:290] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Requested vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:37.771704 7742 raft_consensus.cc:491] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:37.772598 7009 leader_election.cc:304] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters: 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:37.773137 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet 2d815600b288406587e86d2643fa7a69 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:37.772629 7742 raft_consensus.cc:513] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.780793 7716 tablet_bootstrap.cc:492] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:37.781226 7716 ts_tablet_manager.cc:1397] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.004s
I20250626 19:58:37.777061 7863 raft_consensus.cc:2802] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:37.781754 7863 raft_consensus.cc:695] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Becoming Leader. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Running, Role: LEADER
W20250626 19:58:37.783099 6702 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 60a00cf0b1854a20b54c4ae8a59ede29 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 60a00cf0b1854a20b54c4ae8a59ede29 consensus state has no leader
I20250626 19:58:37.783622 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53819554bad8423fbaf39567acadfeda" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:37.784500 7781 raft_consensus.cc:491] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:37.784196 7716 raft_consensus.cc:357] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.784922 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 17d9cd26171943088ad676722edf9577 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:37.784967 7781 raft_consensus.cc:513] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:37.786479 7568 tablet_bootstrap.cc:654] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.787937 7781 leader_election.cc:290] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:37.787575 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e95e6754e74a491087c4ea48e06c1e97" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:37.789477 7071 raft_consensus.cc:2466] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:37.790297 7569 ts_tablet_manager.cc:1428] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.058s user 0.011s sys 0.000s
I20250626 19:58:37.790511 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e95e6754e74a491087c4ea48e06c1e97" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
W20250626 19:58:37.787899 7006 leader_election.cc:343] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:37.791106 7216 raft_consensus.cc:2466] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:37.793149 7863 consensus_queue.cc:237] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.784987 7716 raft_consensus.cc:383] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.795631 7742 leader_election.cc:290] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:37.796203 7716 raft_consensus.cc:738] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.796793 7560 ts_tablet_manager.cc:1918] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:37.798273 7560 log.cc:1199] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/8ce27ec457e246d8bb4f693934932810
I20250626 19:58:37.790627 7294 leader_election.cc:304] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:37.799185 7716 consensus_queue.cc:260] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.801103 7560 ts_tablet_manager.cc:1939] T 8ce27ec457e246d8bb4f693934932810 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:37.800657 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53819554bad8423fbaf39567acadfeda" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
W20250626 19:58:37.802258 7009 leader_election.cc:343] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:37.802690 7009 leader_election.cc:304] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:37.803967 7742 raft_consensus.cc:2747] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:37.800246 7568 tablet_bootstrap.cc:492] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:37.808574 7568 ts_tablet_manager.cc:1397] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.000s
I20250626 19:58:37.815409 7781 raft_consensus.cc:2802] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:37.815757 7781 raft_consensus.cc:491] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:37.816068 7781 raft_consensus.cc:3058] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:37.817298 7473 ts_tablet_manager.cc:1918] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:37.817632 7473 log.cc:1199] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/17d9cd26171943088ad676722edf9577
I20250626 19:58:37.818250 7473 ts_tablet_manager.cc:1939] T 17d9cd26171943088ad676722edf9577 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:37.812915 7568 raft_consensus.cc:357] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.825016 7568 raft_consensus.cc:383] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.825655 7568 raft_consensus.cc:738] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.826355 7568 consensus_queue.cc:260] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.828348 7568 ts_tablet_manager.cc:1428] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20250626 19:58:37.828503 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 17d9cd26171943088ad676722edf9577 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:37.828302 7781 raft_consensus.cc:513] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:37.831058 7568 tablet_bootstrap.cc:492] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:37.831054 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e95e6754e74a491087c4ea48e06c1e97" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:37.831650 7071 raft_consensus.cc:3058] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:37.833776 7473 tablet_replica.cc:331] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:37.837487 7473 raft_consensus.cc:2241] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:37.837958 7473 raft_consensus.cc:2270] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:37.839108 7071 raft_consensus.cc:2466] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:37.843581 7294 leader_election.cc:304] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:37.844621 7849 raft_consensus.cc:2802] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:37.844843 7473 ts_tablet_manager.cc:1905] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:37.852138 7716 ts_tablet_manager.cc:1428] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.071s user 0.007s sys 0.001s
I20250626 19:58:37.855592 7781 leader_election.cc:290] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:37.856832 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e95e6754e74a491087c4ea48e06c1e97" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:37.857410 7216 raft_consensus.cc:3058] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:37.858891 7568 tablet_bootstrap.cc:654] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.861595 6711 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 2d815600b288406587e86d2643fa7a69
I20250626 19:58:37.863595 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d815600b288406587e86d2643fa7a69 with delete_type TABLET_DATA_DELETED (replaced by tablet 089e49e29b2b4f3dbea8a4597d9e36a2) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:37.863487 7716 tablet_bootstrap.cc:492] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:37.865038 7559 tablet_replica.cc:331] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:37.865577 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d815600b288406587e86d2643fa7a69 with delete_type TABLET_DATA_DELETED (replaced by tablet 089e49e29b2b4f3dbea8a4597d9e36a2) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:37.865746 7559 raft_consensus.cc:2241] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:37.866194 7559 raft_consensus.cc:2270] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:37.866916 6711 catalog_manager.cc:6652] ReplaceTablet: tablet 2d815600b288406587e86d2643fa7a69 deleted and replaced by tablet 089e49e29b2b4f3dbea8a4597d9e36a2
I20250626 19:58:37.867655 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d815600b288406587e86d2643fa7a69 with delete_type TABLET_DATA_DELETED (replaced by tablet 089e49e29b2b4f3dbea8a4597d9e36a2) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:37.869791 7559 ts_tablet_manager.cc:1905] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:37.872836 7776 raft_consensus.cc:491] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:37.873322 7776 raft_consensus.cc:513] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.876034 7776 leader_election.cc:290] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:37.880846 6716 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 53819554bad8423fbaf39567acadfeda (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
W20250626 19:58:37.882285 6716 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 53819554bad8423fbaf39567acadfeda (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 53819554bad8423fbaf39567acadfeda consensus state has no leader
I20250626 19:58:37.882201 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 55fad5c8639047a0a5519243d6ae76a3 (DEFAULT_TABLE table=default.table_6bf87f28d4d34102bf1d02e2e1465a33 [id=58d985607f30440ea82dab74e20a1f8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.883291 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55fad5c8639047a0a5519243d6ae76a3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:58:37.883221 7152 leader_election.cc:343] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: 17d9cd26171943088ad676722edf9577
I20250626 19:58:37.886819 7216 raft_consensus.cc:2466] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:37.892582 6716 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 53819554bad8423fbaf39567acadfeda (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]). Expected version 1 got 0
I20250626 19:58:37.893491 6716 catalog_manager.cc:5582] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca reported cstate change: term changed from 0 to 1, leader changed from <none> to bbe151be01e24c82854d9c8c79c797ca (127.4.46.129). New cstate: current_term: 1 leader_uuid: "bbe151be01e24c82854d9c8c79c797ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:37.894852 6716 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 60a00cf0b1854a20b54c4ae8a59ede29 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
I20250626 19:58:37.894716 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 55fad5c8639047a0a5519243d6ae76a3 (DEFAULT_TABLE table=default.table_6bf87f28d4d34102bf1d02e2e1465a33 [id=58d985607f30440ea82dab74e20a1f8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.895711 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55fad5c8639047a0a5519243d6ae76a3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:58:37.896126 7150 leader_election.cc:343] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Not found: Tablet not found: 17d9cd26171943088ad676722edf9577
I20250626 19:58:37.896449 7150 leader_election.cc:304] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a; no voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:37.897429 7776 raft_consensus.cc:2747] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:37.898110 7849 raft_consensus.cc:695] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:37.898985 7849 consensus_queue.cc:237] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:37.908763 6709 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_acc449d45fc742dab0b039b077cc75e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:37.921181 7568 tablet_bootstrap.cc:492] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:37.921572 7568 ts_tablet_manager.cc:1397] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.091s user 0.008s sys 0.008s
I20250626 19:58:37.923828 7568 raft_consensus.cc:357] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.924427 7568 raft_consensus.cc:383] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.924675 7568 raft_consensus.cc:738] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.932134 7568 consensus_queue.cc:260] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.941812 7559 ts_tablet_manager.cc:1918] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:37.942660 7559 log.cc:1199] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/2d815600b288406587e86d2643fa7a69
I20250626 19:58:37.943794 7568 ts_tablet_manager.cc:1428] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.022s user 0.006s sys 0.002s
I20250626 19:58:37.945034 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 40b6570e38434e0c8301d010e96c67a3 (DEFAULT_TABLE table=default.table_ced1a26248c748fb8c06cd348d12fb19 [id=3f20c2a68db64ff7afbc98716ecf20b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.946116 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40b6570e38434e0c8301d010e96c67a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.946143 7559 ts_tablet_manager.cc:1939] T 2d815600b288406587e86d2643fa7a69 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:37.950807 7568 tablet_bootstrap.cc:492] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:37.948554 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 2d815600b288406587e86d2643fa7a69 (table default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]) successfully deleted
I20250626 19:58:37.954622 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 55fad5c8639047a0a5519243d6ae76a3 (DEFAULT_TABLE table=default.table_6bf87f28d4d34102bf1d02e2e1465a33 [id=58d985607f30440ea82dab74e20a1f8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.955219 7716 tablet_bootstrap.cc:654] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.955780 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55fad5c8639047a0a5519243d6ae76a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.958148 7473 ts_tablet_manager.cc:1918] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:37.957967 7776 consensus_queue.cc:1035] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:37.958467 7473 log.cc:1199] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/8ce27ec457e246d8bb4f693934932810
I20250626 19:58:37.927457 6716 master_service.cc:948] ReplaceTablet: received request to replace tablet edf78b9b864446f08056f25423942db3 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:37.970971 7568 tablet_bootstrap.cc:654] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:37.972453 7473 ts_tablet_manager.cc:1939] T 8ce27ec457e246d8bb4f693934932810 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:37.974910 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 40b6570e38434e0c8301d010e96c67a3 (DEFAULT_TABLE table=default.table_ced1a26248c748fb8c06cd348d12fb19 [id=3f20c2a68db64ff7afbc98716ecf20b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:37.975958 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40b6570e38434e0c8301d010e96c67a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:37.977144 7473 tablet_replica.cc:331] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:37.977777 7473 raft_consensus.cc:2241] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:37.978551 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 8ce27ec457e246d8bb4f693934932810 (table default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]) successfully deleted
W20250626 19:58:37.979486 6705 catalog_manager.cc:4895] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 17d9cd26171943088ad676722edf9577 because the tablet was not found. No further retry: Not found: Tablet not found: 17d9cd26171943088ad676722edf9577
I20250626 19:58:37.978181 7473 raft_consensus.cc:2270] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:37.987980 7473 ts_tablet_manager.cc:1905] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:37.991302 7716 tablet_bootstrap.cc:492] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:37.991693 7716 ts_tablet_manager.cc:1397] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.128s user 0.008s sys 0.003s
I20250626 19:58:37.993827 7716 raft_consensus.cc:357] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:37.994418 7716 raft_consensus.cc:383] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:37.994652 7716 raft_consensus.cc:738] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:37.995412 7716 consensus_queue.cc:260] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.005275 7716 ts_tablet_manager.cc:1428] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20250626 19:58:38.017418 6711 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 60a00cf0b1854a20b54c4ae8a59ede29 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
I20250626 19:58:38.020174 7340 tablet_service.cc:1468] Processing CreateTablet for tablet 40b6570e38434e0c8301d010e96c67a3 (DEFAULT_TABLE table=default.table_ced1a26248c748fb8c06cd348d12fb19 [id=3f20c2a68db64ff7afbc98716ecf20b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.021275 7340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40b6570e38434e0c8301d010e96c67a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.029521 7560 tablet_replica.cc:331] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:38.029968 7849 raft_consensus.cc:491] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:38.030362 7849 raft_consensus.cc:513] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.032053 7849 leader_election.cc:290] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:38.033593 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53819554bad8423fbaf39567acadfeda" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:38.034370 7071 raft_consensus.cc:2466] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:38.035099 7052 tablet_service.cc:1468] Processing CreateTablet for tablet fabf0275e32e400c9731eb09d9c2620c (DEFAULT_TABLE table=default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.035444 7294 leader_election.cc:304] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:38.036047 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fabf0275e32e400c9731eb09d9c2620c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.036188 7849 raft_consensus.cc:2802] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:38.036487 7849 raft_consensus.cc:491] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:38.036751 7849 raft_consensus.cc:3058] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:38.043264 7849 raft_consensus.cc:513] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.045958 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53819554bad8423fbaf39567acadfeda" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:38.046536 7071 raft_consensus.cc:3058] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:38.033108 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53819554bad8423fbaf39567acadfeda" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
W20250626 19:58:38.052103 7293 leader_election.cc:343] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:38.053378 7560 raft_consensus.cc:2241] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:38.053864 7560 raft_consensus.cc:2270] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:38.054693 7071 raft_consensus.cc:2466] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:38.055824 7294 leader_election.cc:304] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:38.056716 7867 raft_consensus.cc:2802] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:38.057145 7867 raft_consensus.cc:695] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:38.057528 7341 tablet_service.cc:1468] Processing CreateTablet for tablet fabf0275e32e400c9731eb09d9c2620c (DEFAULT_TABLE table=default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.057874 7867 consensus_queue.cc:237] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.060444 7341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fabf0275e32e400c9731eb09d9c2620c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.068917 7776 raft_consensus.cc:491] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:38.069373 7776 raft_consensus.cc:513] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.071161 7776 leader_election.cc:290] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:38.071537 7776 raft_consensus.cc:491] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:38.071895 7776 raft_consensus.cc:513] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.073421 7776 leader_election.cc:290] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:38.074592 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53819554bad8423fbaf39567acadfeda" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
W20250626 19:58:38.075812 7293 leader_election.cc:343] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:38.076716 7849 leader_election.cc:290] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:38.048442 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 8ce27ec457e246d8bb4f693934932810 (table default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]) successfully deleted
I20250626 19:58:38.080826 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7bf0c724d4684773b40aff24524e1e92" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:38.081409 7073 raft_consensus.cc:2466] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:38.082700 7195 tablet_service.cc:1468] Processing CreateTablet for tablet fabf0275e32e400c9731eb09d9c2620c (DEFAULT_TABLE table=default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.083817 7195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fabf0275e32e400c9731eb09d9c2620c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.086581 7150 leader_election.cc:304] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:38.088279 7776 raft_consensus.cc:2802] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:38.089020 7776 raft_consensus.cc:491] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:38.089665 7776 raft_consensus.cc:3058] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:38.090530 6718 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]
I20250626 19:58:38.092157 7473 ts_tablet_manager.cc:1918] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250626 19:58:38.097152 6702 catalog_manager.cc:5001] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): alter failed for tablet 38b99ee757714ee0a3cfe0a22f476d81 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]),no further retry: Not found: Tablet not found: 38b99ee757714ee0a3cfe0a22f476d81
I20250626 19:58:38.105665 7776 raft_consensus.cc:513] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.078555 6716 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet edf78b9b864446f08056f25423942db3
I20250626 19:58:38.093190 7361 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e267bd75d7d342a3a6c28b17eb37d83e" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:38.111251 7473 log.cc:1199] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/2d815600b288406587e86d2643fa7a69
I20250626 19:58:38.093840 7359 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7bf0c724d4684773b40aff24524e1e92" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:38.096611 7716 tablet_bootstrap.cc:492] T 40b6570e38434e0c8301d010e96c67a3 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:38.111899 7473 ts_tablet_manager.cc:1939] T 2d815600b288406587e86d2643fa7a69 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:38.111866 7049 tablet_service.cc:1515] Processing DeleteTablet for tablet edf78b9b864446f08056f25423942db3 with delete_type TABLET_DATA_DELETED (replaced by tablet 83493d647cc8464e8e650bbca7c6072c) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:38.117113 7559 tablet_replica.cc:331] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:38.117841 7559 raft_consensus.cc:2241] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:38.118307 7559 raft_consensus.cc:2270] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:38.119925 7559 ts_tablet_manager.cc:1905] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:38.124343 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 2d815600b288406587e86d2643fa7a69 (table default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]) successfully deleted
I20250626 19:58:38.104633 7867 raft_consensus.cc:491] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:38.125102 7867 raft_consensus.cc:513] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.126055 7361 raft_consensus.cc:2466] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:38.127195 7867 leader_election.cc:290] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:38.106251 7849 raft_consensus.cc:491] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:38.127811 7849 raft_consensus.cc:513] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.128818 7359 raft_consensus.cc:2466] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:38.129653 7849 leader_election.cc:290] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:38.131983 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7bf0c724d4684773b40aff24524e1e92" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:38.132619 7073 raft_consensus.cc:2466] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:38.134052 7294 leader_election.cc:304] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:38.134474 7716 tablet_bootstrap.cc:654] T 40b6570e38434e0c8301d010e96c67a3 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:38.135058 7849 raft_consensus.cc:2802] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:38.135341 7849 raft_consensus.cc:491] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:38.135597 7849 raft_consensus.cc:3058] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:38.138060 7294 leader_election.cc:343] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e
I20250626 19:58:38.141773 6716 catalog_manager.cc:6652] ReplaceTablet: tablet edf78b9b864446f08056f25423942db3 deleted and replaced by tablet 83493d647cc8464e8e650bbca7c6072c
I20250626 19:58:38.149703 6716 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 60a00cf0b1854a20b54c4ae8a59ede29 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
I20250626 19:58:38.150362 7340 tablet_service.cc:1515] Processing DeleteTablet for tablet edf78b9b864446f08056f25423942db3 with delete_type TABLET_DATA_DELETED (replaced by tablet 83493d647cc8464e8e650bbca7c6072c) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:38.151747 7473 tablet_replica.cc:331] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:38.152443 7473 raft_consensus.cc:2241] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:38.152938 7473 raft_consensus.cc:2270] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:38.119903 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet edf78b9b864446f08056f25423942db3 with delete_type TABLET_DATA_DELETED (replaced by tablet 83493d647cc8464e8e650bbca7c6072c) from {username='slave'} at 127.0.0.1:48028
W20250626 19:58:38.124471 7150 leader_election.cc:343] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e
I20250626 19:58:38.131796 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e267bd75d7d342a3a6c28b17eb37d83e" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:38.155139 7216 raft_consensus.cc:2466] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:38.132443 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7bf0c724d4684773b40aff24524e1e92" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:38.155691 7152 leader_election.cc:304] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:38.156427 7876 raft_consensus.cc:2802] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:38.156504 7293 leader_election.cc:304] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:38.156775 7876 raft_consensus.cc:491] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:38.157053 7876 raft_consensus.cc:3058] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:38.157829 7867 raft_consensus.cc:2802] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:38.158145 7867 raft_consensus.cc:491] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:38.158432 7867 raft_consensus.cc:3058] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:38.159410 7716 tablet_bootstrap.cc:492] T 40b6570e38434e0c8301d010e96c67a3 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:38.159878 7716 ts_tablet_manager.cc:1397] T 40b6570e38434e0c8301d010e96c67a3 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.004s
I20250626 19:58:38.165130 7716 raft_consensus.cc:357] T 40b6570e38434e0c8301d010e96c67a3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.165201 7867 raft_consensus.cc:513] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.165778 7716 raft_consensus.cc:383] T 40b6570e38434e0c8301d010e96c67a3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:38.166042 7716 raft_consensus.cc:738] T 40b6570e38434e0c8301d010e96c67a3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:38.166131 7876 raft_consensus.cc:513] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.167282 7473 ts_tablet_manager.cc:1905] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:38.169320 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7bf0c724d4684773b40aff24524e1e92" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:38.169986 7071 raft_consensus.cc:3058] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:38.150776 7849 raft_consensus.cc:513] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.172885 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7bf0c724d4684773b40aff24524e1e92" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:38.144724 7560 ts_tablet_manager.cc:1905] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:38.167045 7215 raft_consensus.cc:2391] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 297d2c913667408bae82668a557e06c7 in current term 1: Already voted for candidate 315d304854c541bf8b5e6b1eea712d4a in this term.
I20250626 19:58:38.174568 7876 leader_election.cc:290] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:38.174799 7849 leader_election.cc:290] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:38.179025 7716 consensus_queue.cc:260] T 40b6570e38434e0c8301d010e96c67a3 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.144574 7568 tablet_bootstrap.cc:492] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
W20250626 19:58:38.177446 7150 leader_election.cc:343] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e
I20250626 19:58:38.178002 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7bf0c724d4684773b40aff24524e1e92" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:38.180476 7215 raft_consensus.cc:2391] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 297d2c913667408bae82668a557e06c7 in current term 1: Already voted for candidate 315d304854c541bf8b5e6b1eea712d4a in this term.
I20250626 19:58:38.181061 7716 ts_tablet_manager.cc:1428] T 40b6570e38434e0c8301d010e96c67a3 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20250626 19:58:38.181484 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e267bd75d7d342a3a6c28b17eb37d83e" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
W20250626 19:58:38.181731 7294 leader_election.cc:343] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e
I20250626 19:58:38.182098 7360 raft_consensus.cc:2391] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 315d304854c541bf8b5e6b1eea712d4a in current term 1: Already voted for candidate 297d2c913667408bae82668a557e06c7 in this term.
I20250626 19:58:38.182515 7293 leader_election.cc:304] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 297d2c913667408bae82668a557e06c7; no voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:38.183104 7152 leader_election.cc:304] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a; no voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:38.183769 7781 raft_consensus.cc:2747] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:58:38.184707 7568 ts_tablet_manager.cc:1397] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.234s user 0.004s sys 0.022s
I20250626 19:58:38.185927 7360 raft_consensus.cc:1273] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Refusing update from remote peer bbe151be01e24c82854d9c8c79c797ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:58:38.187157 7568 raft_consensus.cc:357] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.187227 7863 consensus_queue.cc:1035] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:38.187841 7568 raft_consensus.cc:383] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:38.188154 7568 raft_consensus.cc:738] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:38.188843 7568 consensus_queue.cc:260] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.193287 7215 raft_consensus.cc:3058] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:38.196292 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e267bd75d7d342a3a6c28b17eb37d83e" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:38.197007 7216 raft_consensus.cc:2391] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 297d2c913667408bae82668a557e06c7 in current term 1: Already voted for candidate 315d304854c541bf8b5e6b1eea712d4a in this term.
I20250626 19:58:38.198467 7293 leader_election.cc:304] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 297d2c913667408bae82668a557e06c7; no voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:38.200723 7849 raft_consensus.cc:2747] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:58:38.201094 7215 raft_consensus.cc:1273] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Refusing update from remote peer bbe151be01e24c82854d9c8c79c797ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:58:38.202693 7863 consensus_queue.cc:1035] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:38.206848 7867 leader_election.cc:290] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:38.211519 7716 tablet_bootstrap.cc:492] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:38.211819 7071 raft_consensus.cc:2466] T 7bf0c724d4684773b40aff24524e1e92 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:38.214391 7569 tablet_bootstrap.cc:492] T 55fad5c8639047a0a5519243d6ae76a3 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:38.212783 7150 leader_election.cc:304] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:38.216904 7716 tablet_bootstrap.cc:654] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:38.220223 7569 tablet_bootstrap.cc:654] T 55fad5c8639047a0a5519243d6ae76a3 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:38.222067 7776 leader_election.cc:290] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:38.223294 7359 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7bf0c724d4684773b40aff24524e1e92" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:38.224086 7359 raft_consensus.cc:2391] T 7bf0c724d4684773b40aff24524e1e92 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 315d304854c541bf8b5e6b1eea712d4a in current term 1: Already voted for candidate 297d2c913667408bae82668a557e06c7 in this term.
I20250626 19:58:38.223305 7876 raft_consensus.cc:2802] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:38.225937 7876 raft_consensus.cc:695] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:38.226824 7876 consensus_queue.cc:237] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.211028 7886 raft_consensus.cc:2747] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 19:58:38.233599 7568 ts_tablet_manager.cc:1428] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20250626 19:58:38.236197 7560 ts_tablet_manager.cc:1918] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:38.236582 7560 log.cc:1199] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/17d9cd26171943088ad676722edf9577
I20250626 19:58:38.237231 7560 ts_tablet_manager.cc:1939] T 17d9cd26171943088ad676722edf9577 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:38.238408 7559 ts_tablet_manager.cc:1918] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:38.238819 7559 log.cc:1199] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/edf78b9b864446f08056f25423942db3
I20250626 19:58:38.239434 7559 ts_tablet_manager.cc:1939] T edf78b9b864446f08056f25423942db3 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:38.239733 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 17d9cd26171943088ad676722edf9577 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:38.241833 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet edf78b9b864446f08056f25423942db3 (table default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]) successfully deleted
I20250626 19:58:38.247380 7568 tablet_bootstrap.cc:492] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:38.248095 7890 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.46.189:36417 every 8 connection(s)
I20250626 19:58:38.251948 7560 tablet_replica.cc:331] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:38.252871 7560 raft_consensus.cc:2241] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:38.253398 7560 raft_consensus.cc:2270] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:38.255347 7453 rpc_server.cc:307] RPC server started. Bound to: 127.4.46.189:36417
I20250626 19:58:38.255815 7716 tablet_bootstrap.cc:492] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:38.256229 7716 ts_tablet_manager.cc:1397] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.000s
I20250626 19:58:38.260718 7568 tablet_bootstrap.cc:654] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:38.271127 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:58:38.266594 6716 master_service.cc:948] ReplaceTablet: received request to replace tablet 578cf59de39b427d8a3074d30e3aeb84 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:38.272634 7716 raft_consensus.cc:357] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.273252 7716 raft_consensus.cc:383] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:38.273510 7716 raft_consensus.cc:738] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:38.274817 7569 tablet_bootstrap.cc:492] T 55fad5c8639047a0a5519243d6ae76a3 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:38.280715 7560 ts_tablet_manager.cc:1905] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:38.285476 7716 consensus_queue.cc:260] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.288064 7716 ts_tablet_manager.cc:1428] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20250626 19:58:38.288065 7569 ts_tablet_manager.cc:1397] T 55fad5c8639047a0a5519243d6ae76a3 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.074s user 0.013s sys 0.000s
I20250626 19:58:38.288865 7716 tablet_bootstrap.cc:492] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:38.290699 7569 raft_consensus.cc:357] T 55fad5c8639047a0a5519243d6ae76a3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.290688 7899 tablet.cc:1722] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:58:38.292232 7473 ts_tablet_manager.cc:1918] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:38.292562 7473 log.cc:1199] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/edf78b9b864446f08056f25423942db3
I20250626 19:58:38.293165 7473 ts_tablet_manager.cc:1939] T edf78b9b864446f08056f25423942db3 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:38.294409 7453 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/master-1/data/info.pb
I20250626 19:58:38.297089 6712 catalog_manager.cc:2149] created HMS catalog entry for table [id=853420b7ab29422e9683dd9e353f87af]
I20250626 19:58:38.300684 4282 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskAJMAmr/build/tsan/bin/kudu as pid 7453
I20250626 19:58:38.318210 6944 catalog_manager.cc:2747] Deleting table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8] in response to Hive Metastore notification log event 17
I20250626 19:58:38.321233 7716 tablet_bootstrap.cc:654] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:38.339617 6712 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 80) took 2944 ms. Trace:
I20250626 19:58:38.339998 6712 rpcz_store.cc:276] 0626 19:58:35.395567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:35.395692 (+ 125us) service_pool.cc:225] Handling call
0626 19:58:36.394029 (+998337us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:36.401656 (+ 7627us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:38.297049 (+1895393us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:38.338893 (+ 41844us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:38.339219 (+ 326us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:38.339551 (+ 332us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:38.321695 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:38.322103 (+ 408us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:38.322126 (+ 23us) write_op.cc:435] Acquired schema lock
0626 19:58:38.322142 (+ 16us) tablet.cc:660] Decoding operations
0626 19:58:38.322458 (+ 316us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:38.322494 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:38.322513 (+ 19us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:38.322707 (+ 194us) tablet.cc:699] Row locks acquired
0626 19:58:38.322719 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:58:38.322850 (+ 131us) write_op.cc:270] Start()
0626 19:58:38.322931 (+ 81us) write_op.cc:276] Timestamp: P: 1750967918322858 usec, L: 0
0626 19:58:38.322946 (+ 15us) op_driver.cc:348] REPLICATION: starting
0626 19:58:38.323433 (+ 487us) log.cc:844] Serialized 433 byte log entry
0626 19:58:38.336912 (+ 13479us) op_driver.cc:464] REPLICATION: finished
0626 19:58:38.337280 (+ 368us) write_op.cc:301] APPLY: starting
0626 19:58:38.337329 (+ 49us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:38.337418 (+ 89us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:38.337498 (+ 80us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:38.337834 (+ 336us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:38.337952 (+ 118us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:38.337988 (+ 36us) write_op.cc:312] APPLY: finished
0626 19:58:38.338136 (+ 148us) log.cc:844] Serialized 29 byte log entry
0626 19:58:38.338265 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:38.338330 (+ 65us) write_op.cc:454] Released schema lock
0626 19:58:38.338363 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0626 19:58:38.338592 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:38.338614 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1688425,"Hive Metastore.run_cpu_time_us":628,"Hive Metastore.run_wall_time_us":206105,"child_traces":[["op",{"apply.queue_time_us":232,"apply.run_cpu_time_us":1278,"apply.run_wall_time_us":1278,"mutex_wait_us":2942,"num_ops":2,"prepare.queue_time_us":22720,"prepare.run_cpu_time_us":2093,"prepare.run_wall_time_us":2278,"raft.queue_time_us":110,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":15,"replication_time_us":13884}]]}
W20250626 19:58:38.349442 6712 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
I20250626 19:58:38.351444 7716 tablet_bootstrap.cc:492] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:38.351822 7716 ts_tablet_manager.cc:1397] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.001s
I20250626 19:58:38.354053 7716 raft_consensus.cc:357] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.354909 7560 ts_tablet_manager.cc:1918] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:38.355268 7560 log.cc:1199] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/2d815600b288406587e86d2643fa7a69
W20250626 19:58:38.349934 6712 catalog_manager.cc:5415] Ignoring report from unknown tablet 2d815600b288406587e86d2643fa7a69
I20250626 19:58:38.354666 7716 raft_consensus.cc:383] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:38.356227 7716 raft_consensus.cc:738] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:38.355922 7560 ts_tablet_manager.cc:1939] T 2d815600b288406587e86d2643fa7a69 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:38.358992 7716 consensus_queue.cc:260] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.359733 7568 tablet_bootstrap.cc:492] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:38.360116 7568 ts_tablet_manager.cc:1397] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.113s user 0.008s sys 0.000s
I20250626 19:58:38.362735 6716 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 578cf59de39b427d8a3074d30e3aeb84
I20250626 19:58:38.363922 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet edf78b9b864446f08056f25423942db3 (table default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]) successfully deleted
I20250626 19:58:38.365199 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 578cf59de39b427d8a3074d30e3aeb84 with delete_type TABLET_DATA_DELETED (replaced by tablet c7159554b45442b5aa5069d7a47d1f8e) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:38.334677 7897 tablet.cc:1722] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:58:38.366003 7569 raft_consensus.cc:383] T 55fad5c8639047a0a5519243d6ae76a3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:38.372380 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 578cf59de39b427d8a3074d30e3aeb84 with delete_type TABLET_DATA_DELETED (replaced by tablet c7159554b45442b5aa5069d7a47d1f8e) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:38.373171 7473 tablet_replica.cc:331] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:38.374336 7716 ts_tablet_manager.cc:1428] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20250626 19:58:38.373476 7569 raft_consensus.cc:738] T 55fad5c8639047a0a5519243d6ae76a3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:38.375204 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 578cf59de39b427d8a3074d30e3aeb84 with delete_type TABLET_DATA_DELETED (replaced by tablet c7159554b45442b5aa5069d7a47d1f8e) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:38.376451 7560 tablet_replica.cc:331] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:38.377177 7560 raft_consensus.cc:2241] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:38.377619 7560 raft_consensus.cc:2270] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:38.378993 7559 tablet_replica.cc:331] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:38.379627 7559 raft_consensus.cc:2241] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:38.379294 7560 ts_tablet_manager.cc:1905] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:38.380064 7559 raft_consensus.cc:2270] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:38.377377 6716 catalog_manager.cc:6652] ReplaceTablet: tablet 578cf59de39b427d8a3074d30e3aeb84 deleted and replaced by tablet c7159554b45442b5aa5069d7a47d1f8e
I20250626 19:58:38.381651 7559 ts_tablet_manager.cc:1905] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:38.383069 7867 raft_consensus.cc:491] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:38.383492 7867 raft_consensus.cc:513] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.385375 7867 leader_election.cc:290] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:38.380496 7569 consensus_queue.cc:260] T 55fad5c8639047a0a5519243d6ae76a3 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.387060 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fabf0275e32e400c9731eb09d9c2620c" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
W20250626 19:58:38.388533 7293 leader_election.cc:343] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:38.388792 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fabf0275e32e400c9731eb09d9c2620c" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
W20250626 19:58:38.390210 7294 leader_election.cc:343] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:38.390596 7294 leader_election.cc:304] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 297d2c913667408bae82668a557e06c7; no voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:38.391625 7867 raft_consensus.cc:2747] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:38.398141 7569 ts_tablet_manager.cc:1428] T 55fad5c8639047a0a5519243d6ae76a3 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.109s user 0.007s sys 0.000s
I20250626 19:58:38.405403 6716 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 60a00cf0b1854a20b54c4ae8a59ede29 (table default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb]). Expected version 1 got 0
I20250626 19:58:38.408764 7569 tablet_bootstrap.cc:492] T fabf0275e32e400c9731eb09d9c2620c P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:38.411293 7473 raft_consensus.cc:2241] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:38.412005 7473 pending_rounds.cc:70] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7: Trying to abort 1 pending ops.
I20250626 19:58:38.412216 7473 pending_rounds.cc:77] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964590572089344 op_type: NO_OP noop_request { }
I20250626 19:58:38.412535 7473 raft_consensus.cc:2887] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:38.412940 7473 raft_consensus.cc:2270] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:38.415510 7473 ts_tablet_manager.cc:1905] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:38.417037 7568 raft_consensus.cc:357] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.418009 7568 raft_consensus.cc:383] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:38.418390 7568 raft_consensus.cc:738] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:38.376173 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 2d815600b288406587e86d2643fa7a69 (table default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]) successfully deleted
I20250626 19:58:38.431318 7560 ts_tablet_manager.cc:1918] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:38.435261 7560 log.cc:1199] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/edf78b9b864446f08056f25423942db3
I20250626 19:58:38.435909 7560 ts_tablet_manager.cc:1939] T edf78b9b864446f08056f25423942db3 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:38.435557 7568 consensus_queue.cc:260] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.437958 7568 ts_tablet_manager.cc:1428] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.078s user 0.002s sys 0.005s
I20250626 19:58:38.438452 7560 tablet_replica.cc:331] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:38.438825 7568 tablet_bootstrap.cc:492] T 55fad5c8639047a0a5519243d6ae76a3 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:38.439153 7560 raft_consensus.cc:2241] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:38.439589 7560 raft_consensus.cc:2270] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:38.440053 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet edf78b9b864446f08056f25423942db3 (table default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]) successfully deleted
I20250626 19:58:38.441193 7560 ts_tablet_manager.cc:1905] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:38.448236 7262 heartbeater.cc:344] Connected to a master server at 127.4.46.189:36417
I20250626 19:58:38.454006 7119 heartbeater.cc:344] Connected to a master server at 127.4.46.189:36417
I20250626 19:58:38.460201 7742 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:38.460704 7742 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.460875 7908 tablet.cc:1722] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250626 19:58:38.465412 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 19:58:38.466931 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 19:58:38.467329 7006 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:38.462680 7742 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:38.468233 7861 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:38.469125 7559 ts_tablet_manager.cc:1918] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:38.475867 7559 log.cc:1199] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/578cf59de39b427d8a3074d30e3aeb84
I20250626 19:58:38.476660 7559 ts_tablet_manager.cc:1939] T 578cf59de39b427d8a3074d30e3aeb84 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:38.479091 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 578cf59de39b427d8a3074d30e3aeb84 (table default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a]) successfully deleted
I20250626 19:58:38.490506 7560 ts_tablet_manager.cc:1918] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:38.490891 7560 log.cc:1199] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/578cf59de39b427d8a3074d30e3aeb84
I20250626 19:58:38.491487 7560 ts_tablet_manager.cc:1939] T 578cf59de39b427d8a3074d30e3aeb84 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:38.492174 6711 catalog_manager.cc:5582] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:38.493343 7050 tablet_service.cc:1468] Processing CreateTablet for tablet f9adef69fe7442a4836a84934e792c05 (DEFAULT_TABLE table=default.table_a43e8ff4d1844c0b9dddf28fc310fd8f [id=ef672ba275b54933ae739bf9e560ae68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.494329 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9adef69fe7442a4836a84934e792c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.496330 7051 tablet_service.cc:1468] Processing CreateTablet for tablet 089e49e29b2b4f3dbea8a4597d9e36a2 (DEFAULT_TABLE table=default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.495776 6712 catalog_manager.cc:5582] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130). New cstate: current_term: 1 leader_uuid: "315d304854c541bf8b5e6b1eea712d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:38.497336 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089e49e29b2b4f3dbea8a4597d9e36a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.500922 7194 tablet_service.cc:1468] Processing CreateTablet for tablet f9adef69fe7442a4836a84934e792c05 (DEFAULT_TABLE table=default.table_a43e8ff4d1844c0b9dddf28fc310fd8f [id=ef672ba275b54933ae739bf9e560ae68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.501343 7195 tablet_service.cc:1468] Processing CreateTablet for tablet 089e49e29b2b4f3dbea8a4597d9e36a2 (DEFAULT_TABLE table=default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.501861 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9adef69fe7442a4836a84934e792c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.502395 7195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089e49e29b2b4f3dbea8a4597d9e36a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.504266 7339 tablet_service.cc:1468] Processing CreateTablet for tablet f9adef69fe7442a4836a84934e792c05 (DEFAULT_TABLE table=default.table_a43e8ff4d1844c0b9dddf28fc310fd8f [id=ef672ba275b54933ae739bf9e560ae68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.505218 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9adef69fe7442a4836a84934e792c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.507402 7341 tablet_service.cc:1468] Processing CreateTablet for tablet 089e49e29b2b4f3dbea8a4597d9e36a2 (DEFAULT_TABLE table=default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.508342 7341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089e49e29b2b4f3dbea8a4597d9e36a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.516618 7340 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d815600b288406587e86d2643fa7a69 with delete_type TABLET_DATA_DELETED (replaced by tablet 089e49e29b2b4f3dbea8a4597d9e36a2) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:38.538342 7716 tablet_bootstrap.cc:492] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:38.538650 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 578cf59de39b427d8a3074d30e3aeb84 (table default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a]) successfully deleted
I20250626 19:58:38.543990 7716 tablet_bootstrap.cc:654] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:38.546434 6944 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 53819554bad8423fbaf39567acadfeda
I20250626 19:58:38.549752 7052 tablet_service.cc:1515] Processing DeleteTablet for tablet 53819554bad8423fbaf39567acadfeda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:38 UTC) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:38.550980 7053 tablet_service.cc:1468] Processing CreateTablet for tablet 7b9a0f71535a4a65a775f728edbcc636 (DEFAULT_TABLE table=default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.552335 7053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b9a0f71535a4a65a775f728edbcc636. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.553344 7340 tablet_service.cc:1468] Processing CreateTablet for tablet 7b9a0f71535a4a65a775f728edbcc636 (DEFAULT_TABLE table=default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.554471 7340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b9a0f71535a4a65a775f728edbcc636. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.555168 7559 tablet_replica.cc:331] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:38.555714 7337 tablet_service.cc:1468] Processing CreateTablet for tablet 83493d647cc8464e8e650bbca7c6072c (DEFAULT_TABLE table=default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.556111 7559 raft_consensus.cc:2241] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:38.556628 7559 raft_consensus.cc:2270] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:38.558807 7337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83493d647cc8464e8e650bbca7c6072c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.561030 7052 tablet_service.cc:1468] Processing CreateTablet for tablet 83493d647cc8464e8e650bbca7c6072c (DEFAULT_TABLE table=default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.562256 7052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83493d647cc8464e8e650bbca7c6072c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.565649 7559 ts_tablet_manager.cc:1905] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:38.567072 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet 53819554bad8423fbaf39567acadfeda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:38 UTC) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:38.567744 7560 tablet_replica.cc:331] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:38.568178 7196 tablet_service.cc:1468] Processing CreateTablet for tablet 7b9a0f71535a4a65a775f728edbcc636 (DEFAULT_TABLE table=default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.568378 7560 raft_consensus.cc:2241] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:38.568784 7560 raft_consensus.cc:2270] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:38.569149 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b9a0f71535a4a65a775f728edbcc636. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.571795 7568 tablet_bootstrap.cc:654] T 55fad5c8639047a0a5519243d6ae76a3 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:38.575752 7560 ts_tablet_manager.cc:1905] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:38.578971 7341 tablet_service.cc:1515] Processing DeleteTablet for tablet 53819554bad8423fbaf39567acadfeda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:38 UTC) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:38.586092 7473 ts_tablet_manager.cc:1918] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:38.586444 7193 tablet_service.cc:1468] Processing CreateTablet for tablet 83493d647cc8464e8e650bbca7c6072c (DEFAULT_TABLE table=default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.587420 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83493d647cc8464e8e650bbca7c6072c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.593271 7716 tablet_bootstrap.cc:492] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:38.594152 7716 ts_tablet_manager.cc:1397] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.000s
I20250626 19:58:38.596621 7716 raft_consensus.cc:357] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.606977 7716 raft_consensus.cc:383] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:38.607254 7716 raft_consensus.cc:738] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:38.607853 7473 log.cc:1199] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/578cf59de39b427d8a3074d30e3aeb84
I20250626 19:58:38.611174 7716 consensus_queue.cc:260] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.612004 7473 ts_tablet_manager.cc:1939] T 578cf59de39b427d8a3074d30e3aeb84 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
W20250626 19:58:38.621066 6712 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
I20250626 19:58:38.622539 7568 tablet_bootstrap.cc:492] T 55fad5c8639047a0a5519243d6ae76a3 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:38.614445 7716 ts_tablet_manager.cc:1428] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.020s user 0.001s sys 0.004s
I20250626 19:58:38.623137 7568 ts_tablet_manager.cc:1397] T 55fad5c8639047a0a5519243d6ae76a3 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.185s user 0.012s sys 0.003s
I20250626 19:58:38.625528 7568 raft_consensus.cc:357] T 55fad5c8639047a0a5519243d6ae76a3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.626269 7568 raft_consensus.cc:383] T 55fad5c8639047a0a5519243d6ae76a3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:38.626564 7568 raft_consensus.cc:738] T 55fad5c8639047a0a5519243d6ae76a3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:38.635426 7716 tablet_bootstrap.cc:492] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:38.636957 7473 tablet_replica.cc:331] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:38.637603 7473 raft_consensus.cc:2241] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:38.638171 7473 pending_rounds.cc:70] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7: Trying to abort 1 pending ops.
I20250626 19:58:38.638365 7473 pending_rounds.cc:77] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171964592377024512 op_type: NO_OP noop_request { }
I20250626 19:58:38.638409 7560 ts_tablet_manager.cc:1918] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:38.631424 7568 consensus_queue.cc:260] T 55fad5c8639047a0a5519243d6ae76a3 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.640406 7568 ts_tablet_manager.cc:1428] T 55fad5c8639047a0a5519243d6ae76a3 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20250626 19:58:38.641108 7568 tablet_bootstrap.cc:492] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:38.643524 7560 log.cc:1199] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/53819554bad8423fbaf39567acadfeda
I20250626 19:58:38.638674 7473 raft_consensus.cc:2887] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 19:58:38.647354 7473 raft_consensus.cc:2270] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:38.648558 7716 tablet_bootstrap.cc:654] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:38.626680 7569 tablet_bootstrap.cc:654] T fabf0275e32e400c9731eb09d9c2620c P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:38.649693 7473 ts_tablet_manager.cc:1905] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:38.650267 7867 raft_consensus.cc:491] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:38.650681 7867 raft_consensus.cc:513] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.652385 7867 leader_election.cc:290] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:38.654675 7560 ts_tablet_manager.cc:1939] T 53819554bad8423fbaf39567acadfeda P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:38.654544 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "089e49e29b2b4f3dbea8a4597d9e36a2" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
W20250626 19:58:38.655778 7293 leader_election.cc:343] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:38.657652 7559 ts_tablet_manager.cc:1918] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:38.658036 7559 log.cc:1199] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/53819554bad8423fbaf39567acadfeda
I20250626 19:58:38.663561 7559 ts_tablet_manager.cc:1939] T 53819554bad8423fbaf39567acadfeda P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:38.665517 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 53819554bad8423fbaf39567acadfeda (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
W20250626 19:58:38.682390 6705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.188:38919, user_credentials={real_user=slave}} blocked reactor thread for 59974us
I20250626 19:58:38.684242 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 578cf59de39b427d8a3074d30e3aeb84 (table default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a]) successfully deleted
W20250626 19:58:38.684842 6705 catalog_manager.cc:4895] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 2d815600b288406587e86d2643fa7a69 because the tablet was not found. No further retry: Not found: Tablet not found: 2d815600b288406587e86d2643fa7a69
I20250626 19:58:38.692126 7907 sys_catalog.cc:263] Verifying existing consensus state
I20250626 19:58:38.695500 7568 tablet_bootstrap.cc:654] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:38.698506 7569 tablet_bootstrap.cc:492] T fabf0275e32e400c9731eb09d9c2620c P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:38.702991 7473 ts_tablet_manager.cc:1918] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:38.703365 7473 log.cc:1199] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/53819554bad8423fbaf39567acadfeda
I20250626 19:58:38.704124 7473 ts_tablet_manager.cc:1939] T 53819554bad8423fbaf39567acadfeda P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:38.704972 7716 tablet_bootstrap.cc:492] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:38.705351 7716 ts_tablet_manager.cc:1397] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.070s user 0.011s sys 0.004s
I20250626 19:58:38.707336 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 53819554bad8423fbaf39567acadfeda (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:38.708379 7407 heartbeater.cc:344] Connected to a master server at 127.4.46.189:36417
I20250626 19:58:38.712416 7907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: Bootstrap starting.
I20250626 19:58:38.698621 6711 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: default.table_252448e828024438ba317b8277b66523 [id=ee116b84a24e49228c7496e95eaf11bb] alter complete (version 1)
I20250626 19:58:38.715034 6711 master_service.cc:948] ReplaceTablet: received request to replace tablet 578cf59de39b427d8a3074d30e3aeb84 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:38.716470 6711 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_dc0e9650e86e4c60aec4f82f9b82f959"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:38.721896 7568 tablet_bootstrap.cc:492] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:38.722285 7568 ts_tablet_manager.cc:1397] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.081s user 0.008s sys 0.003s
I20250626 19:58:38.722921 7716 raft_consensus.cc:357] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.723608 7716 raft_consensus.cc:383] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:38.723872 7716 raft_consensus.cc:738] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:38.724426 7716 consensus_queue.cc:260] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.724712 7568 raft_consensus.cc:357] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.725272 7568 raft_consensus.cc:383] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:38.725500 7568 raft_consensus.cc:738] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:38.726063 7568 consensus_queue.cc:260] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.728157 7568 ts_tablet_manager.cc:1428] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250626 19:58:38.728941 7568 tablet_bootstrap.cc:492] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:38.734229 7568 tablet_bootstrap.cc:654] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:38.743299 6716 master_service.cc:948] ReplaceTablet: received request to replace tablet 734d9d3331b44fc49e1f64b8caf1e4d8 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:38.744895 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet 53819554bad8423fbaf39567acadfeda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:38 UTC) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:38.749387 6944 catalog_manager.cc:2747] Deleting table default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c] in response to Hive Metastore notification log event 22
W20250626 19:58:38.749891 6702 catalog_manager.cc:4895] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet 53819554bad8423fbaf39567acadfeda because the tablet was not found. No further retry: Not found: Tablet not found: 53819554bad8423fbaf39567acadfeda
I20250626 19:58:38.752398 7568 tablet_bootstrap.cc:492] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:38.752805 7568 ts_tablet_manager.cc:1397] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.008s
I20250626 19:58:38.755275 7568 raft_consensus.cc:357] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.755877 7568 raft_consensus.cc:383] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:38.756103 7568 raft_consensus.cc:738] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:38.756103 7337 tablet_service.cc:1515] Processing DeleteTablet for tablet 53819554bad8423fbaf39567acadfeda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:38 UTC) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:38.756644 7568 consensus_queue.cc:260] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:38.758117 6705 catalog_manager.cc:4895] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 53819554bad8423fbaf39567acadfeda because the tablet was not found. No further retry: Not found: Tablet not found: 53819554bad8423fbaf39567acadfeda
I20250626 19:58:38.758579 7568 ts_tablet_manager.cc:1428] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250626 19:58:38.764178 7716 ts_tablet_manager.cc:1428] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.059s user 0.002s sys 0.004s
I20250626 19:58:38.765234 7716 tablet_bootstrap.cc:492] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:38.768805 7867 consensus_queue.cc:1035] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:38.776257 7568 tablet_bootstrap.cc:492] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:38.776952 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 53819554bad8423fbaf39567acadfeda (table default.table_a5221df6a6b54bcdafa382fca51b613d [id=eff2225412944029aed88f6c87744fc8]) successfully deleted
I20250626 19:58:38.777540 7193 tablet_service.cc:1468] Processing CreateTablet for tablet c7159554b45442b5aa5069d7a47d1f8e (DEFAULT_TABLE table=default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.778569 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7159554b45442b5aa5069d7a47d1f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.783190 7867 consensus_queue.cc:1035] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:38.784889 7716 tablet_bootstrap.cc:654] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:38.795974 7568 tablet_bootstrap.cc:654] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:38.801359 7337 tablet_service.cc:1468] Processing CreateTablet for tablet c7159554b45442b5aa5069d7a47d1f8e (DEFAULT_TABLE table=default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.802386 7337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7159554b45442b5aa5069d7a47d1f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.809226 7569 ts_tablet_manager.cc:1397] T fabf0275e32e400c9731eb09d9c2620c P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.403s user 0.014s sys 0.019s
I20250626 19:58:38.811235 6712 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 7b9a0f71535a4a65a775f728edbcc636 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
I20250626 19:58:38.811491 7716 tablet_bootstrap.cc:492] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:38.811841 7716 ts_tablet_manager.cc:1397] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.000s
I20250626 19:58:38.811982 7569 raft_consensus.cc:357] T fabf0275e32e400c9731eb09d9c2620c P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.812727 7569 raft_consensus.cc:383] T fabf0275e32e400c9731eb09d9c2620c P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:38.812968 7569 raft_consensus.cc:738] T fabf0275e32e400c9731eb09d9c2620c P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:38.781272 7053 tablet_service.cc:1468] Processing CreateTablet for tablet c7159554b45442b5aa5069d7a47d1f8e (DEFAULT_TABLE table=default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.814312 7053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7159554b45442b5aa5069d7a47d1f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.817690 7716 raft_consensus.cc:357] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.817842 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 9de12a70dc90406c9ca8f58d253f35d5 (DEFAULT_TABLE table=default.table_fa425acbc3844433a95ac1c8ee1dc26d [id=853420b7ab29422e9683dd9e353f87af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.818253 7716 raft_consensus.cc:383] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:38.818468 7716 raft_consensus.cc:738] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:38.818833 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9de12a70dc90406c9ca8f58d253f35d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.781893 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "089e49e29b2b4f3dbea8a4597d9e36a2" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:38.819046 7716 consensus_queue.cc:260] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:38.823324 7294 leader_election.cc:343] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:38.823776 7294 leader_election.cc:304] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 297d2c913667408bae82668a557e06c7; no voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:38.824708 7569 consensus_queue.cc:260] T fabf0275e32e400c9731eb09d9c2620c P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.827216 7569 ts_tablet_manager.cc:1428] T fabf0275e32e400c9731eb09d9c2620c P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20250626 19:58:38.828357 7867 raft_consensus.cc:2747] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:38.830127 7716 ts_tablet_manager.cc:1428] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250626 19:58:38.832512 7716 tablet_bootstrap.cc:492] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:38.839682 7569 tablet_bootstrap.cc:492] T 40b6570e38434e0c8301d010e96c67a3 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:38.842248 7716 tablet_bootstrap.cc:654] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:38.846426 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 9de12a70dc90406c9ca8f58d253f35d5 (DEFAULT_TABLE table=default.table_fa425acbc3844433a95ac1c8ee1dc26d [id=853420b7ab29422e9683dd9e353f87af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.847661 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9de12a70dc90406c9ca8f58d253f35d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.865778 7908 tablet.cc:1722] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250626 19:58:38.866447 6944 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet 089e49e29b2b4f3dbea8a4597d9e36a2
I20250626 19:58:38.868042 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 089e49e29b2b4f3dbea8a4597d9e36a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:38 UTC) from {username='slave'} at 127.0.0.1:54698
W20250626 19:58:38.869737 6703 catalog_manager.cc:4908] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 089e49e29b2b4f3dbea8a4597d9e36a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 089e49e29b2b4f3dbea8a4597d9e36a2 already in progress: creating tablet
I20250626 19:58:38.871380 7196 tablet_service.cc:1515] Processing DeleteTablet for tablet 089e49e29b2b4f3dbea8a4597d9e36a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:38 UTC) from {username='slave'} at 127.0.0.1:48028
W20250626 19:58:38.872766 6702 catalog_manager.cc:4908] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): delete failed for tablet 089e49e29b2b4f3dbea8a4597d9e36a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 089e49e29b2b4f3dbea8a4597d9e36a2 already in progress: creating tablet
I20250626 19:58:38.873142 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 089e49e29b2b4f3dbea8a4597d9e36a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:38 UTC) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:38.873847 7473 tablet_replica.cc:331] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:38.874575 7473 raft_consensus.cc:2241] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:38.875018 7473 raft_consensus.cc:2270] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:38.876999 7473 ts_tablet_manager.cc:1905] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:38.880673 7897 tablet.cc:1722] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250626 19:58:38.893812 7473 ts_tablet_manager.cc:1918] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 19:58:38.894155 7473 log.cc:1199] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/089e49e29b2b4f3dbea8a4597d9e36a2
W20250626 19:58:38.904153 6716 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
W20250626 19:58:38.915722 6712 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 7b9a0f71535a4a65a775f728edbcc636 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 7b9a0f71535a4a65a775f728edbcc636 consensus state has no leader
I20250626 19:58:38.948453 7569 tablet_bootstrap.cc:654] T 40b6570e38434e0c8301d010e96c67a3 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:38.913645 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 9de12a70dc90406c9ca8f58d253f35d5 (DEFAULT_TABLE table=default.table_fa425acbc3844433a95ac1c8ee1dc26d [id=853420b7ab29422e9683dd9e353f87af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:38.947392 7925 raft_consensus.cc:491] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:38.952945 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9de12a70dc90406c9ca8f58d253f35d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:38.953030 7925 raft_consensus.cc:513] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.956001 7925 leader_election.cc:290] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:38.956207 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55fad5c8639047a0a5519243d6ae76a3" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:38.956791 7216 raft_consensus.cc:2466] T 55fad5c8639047a0a5519243d6ae76a3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:38.957892 7293 leader_election.cc:304] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:38.958667 7716 tablet_bootstrap.cc:492] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:38.959128 7716 ts_tablet_manager.cc:1397] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.127s user 0.018s sys 0.006s
I20250626 19:58:38.960681 7867 raft_consensus.cc:2802] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:38.961505 7867 raft_consensus.cc:491] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:38.894753 7473 ts_tablet_manager.cc:1939] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:38.964746 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 089e49e29b2b4f3dbea8a4597d9e36a2 (table default.table_8a5f3c205ce2406e8b57c6f09619cb05 [id=4f3eec76141f4ec38948625110d7136c]) successfully deleted
W20250626 19:58:38.965467 7919 log.cc:927] Time spent T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20250626 19:58:38.962311 7867 raft_consensus.cc:3058] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:38.970909 6703 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 7b9a0f71535a4a65a775f728edbcc636 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7b9a0f71535a4a65a775f728edbcc636 consensus state has no leader
I20250626 19:58:38.983197 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55fad5c8639047a0a5519243d6ae76a3" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:38.983928 7073 raft_consensus.cc:2466] T 55fad5c8639047a0a5519243d6ae76a3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:38.962062 7716 raft_consensus.cc:357] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.987769 7716 raft_consensus.cc:383] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:38.988029 7716 raft_consensus.cc:738] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:38.990459 7867 raft_consensus.cc:513] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:38.993079 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55fad5c8639047a0a5519243d6ae76a3" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:38.993621 7216 raft_consensus.cc:3058] T 55fad5c8639047a0a5519243d6ae76a3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:38.993536 7899 tablet.cc:1722] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250626 19:58:38.995393 7716 consensus_queue.cc:260] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.001003 7216 raft_consensus.cc:2466] T 55fad5c8639047a0a5519243d6ae76a3 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:38.981388 7907 log.cc:826] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: Log is configured to *not* fsync() on all Append() calls
I20250626 19:58:39.002338 7293 leader_election.cc:304] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:39.003209 7925 raft_consensus.cc:2802] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:39.003688 7925 raft_consensus.cc:695] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:39.004494 7925 consensus_queue.cc:237] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.007906 7716 ts_tablet_manager.cc:1428] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.048s user 0.005s sys 0.003s
I20250626 19:58:39.008926 7876 raft_consensus.cc:491] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:39.009327 7876 raft_consensus.cc:513] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.013600 7716 tablet_bootstrap.cc:492] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:39.011011 7876 leader_election.cc:290] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:39.019009 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fabf0275e32e400c9731eb09d9c2620c" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:39.019790 7071 raft_consensus.cc:2466] T fabf0275e32e400c9731eb09d9c2620c P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:39.021819 7150 leader_election.cc:304] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:39.022579 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fabf0275e32e400c9731eb09d9c2620c" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:39.022877 7876 raft_consensus.cc:2802] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:39.023186 7876 raft_consensus.cc:491] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:39.023157 7360 raft_consensus.cc:2466] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:39.023499 7876 raft_consensus.cc:3058] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.030717 7876 raft_consensus.cc:513] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.035771 7568 tablet_bootstrap.cc:492] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:39.036185 7569 tablet_bootstrap.cc:492] T 40b6570e38434e0c8301d010e96c67a3 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:39.036212 7568 ts_tablet_manager.cc:1397] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.260s user 0.010s sys 0.041s
I20250626 19:58:39.036597 7569 ts_tablet_manager.cc:1397] T 40b6570e38434e0c8301d010e96c67a3 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.197s user 0.013s sys 0.006s
I20250626 19:58:39.037379 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fabf0275e32e400c9731eb09d9c2620c" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:39.037902 7073 raft_consensus.cc:3058] T fabf0275e32e400c9731eb09d9c2620c P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.039311 7568 raft_consensus.cc:357] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.039893 7568 raft_consensus.cc:383] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:39.040107 7568 raft_consensus.cc:738] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.040663 7568 consensus_queue.cc:260] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.045720 7073 raft_consensus.cc:2466] T fabf0275e32e400c9731eb09d9c2620c P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:39.042906 7569 raft_consensus.cc:357] T 40b6570e38434e0c8301d010e96c67a3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.046682 7568 ts_tablet_manager.cc:1428] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.010s user 0.004s sys 0.003s
I20250626 19:58:39.046939 7150 leader_election.cc:304] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:39.046854 7569 raft_consensus.cc:383] T 40b6570e38434e0c8301d010e96c67a3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:39.048440 7569 raft_consensus.cc:738] T 40b6570e38434e0c8301d010e96c67a3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.059213 7568 tablet_bootstrap.cc:492] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:39.059183 7932 raft_consensus.cc:2802] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:39.053900 7569 consensus_queue.cc:260] T 40b6570e38434e0c8301d010e96c67a3 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.063781 6716 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: default.table_daf65ec234534a069c2d1ad34f13952a [id=ee116b84a24e49228c7496e95eaf11bb] alter complete (version 2)
I20250626 19:58:39.064842 7568 tablet_bootstrap.cc:654] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:39.067209 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fabf0275e32e400c9731eb09d9c2620c" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:39.067718 7360 raft_consensus.cc:3058] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.061678 7716 tablet_bootstrap.cc:654] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:39.060484 7932 raft_consensus.cc:695] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:39.072686 7867 leader_election.cc:290] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:39.076131 7932 consensus_queue.cc:237] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.077369 7360 raft_consensus.cc:2466] T fabf0275e32e400c9731eb09d9c2620c P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:39.077957 7569 ts_tablet_manager.cc:1428] T 40b6570e38434e0c8301d010e96c67a3 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.041s user 0.005s sys 0.003s
I20250626 19:58:39.078898 7569 tablet_bootstrap.cc:492] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:39.079798 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55fad5c8639047a0a5519243d6ae76a3" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:39.080379 7073 raft_consensus.cc:3058] T 55fad5c8639047a0a5519243d6ae76a3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.081316 7119 heartbeater.cc:344] Connected to a master server at 127.4.46.189:36417
I20250626 19:58:39.092468 7876 leader_election.cc:290] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:39.098248 7568 tablet_bootstrap.cc:492] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
W20250626 19:58:39.098695 7150 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.129:33129, user_credentials={real_user=slave}} blocked reactor thread for 51814.1us
I20250626 19:58:39.098722 7568 ts_tablet_manager.cc:1397] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.000s
I20250626 19:58:39.103664 7568 raft_consensus.cc:357] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.104475 7568 raft_consensus.cc:383] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:39.104782 7568 raft_consensus.cc:738] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.109297 7568 consensus_queue.cc:260] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.110878 7925 raft_consensus.cc:491] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:39.111254 7569 tablet_bootstrap.cc:654] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:39.111410 7925 raft_consensus.cc:513] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.113678 7925 leader_election.cc:290] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:39.115772 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9adef69fe7442a4836a84934e792c05" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
W20250626 19:58:39.117873 7294 leader_election.cc:343] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:39.119064 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9adef69fe7442a4836a84934e792c05" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:39.119541 7568 ts_tablet_manager.cc:1428] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20250626 19:58:39.119707 7216 raft_consensus.cc:2466] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:39.120354 7568 tablet_bootstrap.cc:492] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:39.120774 7293 leader_election.cc:304] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:39.134980 7568 tablet_bootstrap.cc:654] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:39.140079 7925 raft_consensus.cc:2802] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:39.140484 7925 raft_consensus.cc:491] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:39.140847 7925 raft_consensus.cc:3058] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.135761 7932 consensus_queue.cc:1035] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:39.154915 7262 heartbeater.cc:344] Connected to a master server at 127.4.46.189:36417
I20250626 19:58:39.155347 7716 tablet_bootstrap.cc:492] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:39.155764 7716 ts_tablet_manager.cc:1397] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.144s user 0.009s sys 0.006s
I20250626 19:58:39.156284 7932 raft_consensus.cc:491] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:39.156736 7932 raft_consensus.cc:513] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.158563 7932 leader_election.cc:290] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:39.159615 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40b6570e38434e0c8301d010e96c67a3" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:39.160128 7360 raft_consensus.cc:2466] T 40b6570e38434e0c8301d010e96c67a3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:39.161273 7072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40b6570e38434e0c8301d010e96c67a3" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:39.161856 7072 raft_consensus.cc:2466] T 40b6570e38434e0c8301d010e96c67a3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:39.163657 7152 leader_election.cc:304] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:39.164778 7932 raft_consensus.cc:2802] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:39.165077 7932 raft_consensus.cc:491] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:39.165365 7932 raft_consensus.cc:3058] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.158123 7716 raft_consensus.cc:357] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.173012 7716 raft_consensus.cc:383] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:39.173321 7716 raft_consensus.cc:738] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.174017 7716 consensus_queue.cc:260] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.176144 7716 ts_tablet_manager.cc:1428] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20250626 19:58:39.179939 7716 tablet_bootstrap.cc:492] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:39.183660 7925 raft_consensus.cc:513] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.185142 7932 raft_consensus.cc:513] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.188015 7932 leader_election.cc:290] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:39.189206 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40b6570e38434e0c8301d010e96c67a3" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:39.189800 7071 raft_consensus.cc:3058] T 40b6570e38434e0c8301d010e96c67a3 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.191634 7359 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40b6570e38434e0c8301d010e96c67a3" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:39.192174 7359 raft_consensus.cc:3058] T 40b6570e38434e0c8301d010e96c67a3 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.197604 7716 tablet_bootstrap.cc:654] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:39.201611 7359 raft_consensus.cc:2466] T 40b6570e38434e0c8301d010e96c67a3 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:39.206058 7071 raft_consensus.cc:2466] T 40b6570e38434e0c8301d010e96c67a3 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:39.207070 7150 leader_election.cc:304] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:39.207789 7932 raft_consensus.cc:2802] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:39.208210 7932 raft_consensus.cc:695] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:39.208498 7876 consensus_queue.cc:1035] T 7bf0c724d4684773b40aff24524e1e92 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:39.208933 7932 consensus_queue.cc:237] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.209743 7073 raft_consensus.cc:2466] T 55fad5c8639047a0a5519243d6ae76a3 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
W20250626 19:58:39.216053 6712 catalog_manager.cc:5415] Ignoring report from unknown tablet 089e49e29b2b4f3dbea8a4597d9e36a2
I20250626 19:58:39.223142 7569 tablet_bootstrap.cc:492] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:39.223671 7569 ts_tablet_manager.cc:1397] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.145s user 0.018s sys 0.002s
I20250626 19:58:39.224740 7716 tablet_bootstrap.cc:492] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:39.225153 7716 ts_tablet_manager.cc:1397] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.003s
I20250626 19:58:39.226285 7569 raft_consensus.cc:357] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.226977 7569 raft_consensus.cc:383] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:39.227270 7569 raft_consensus.cc:738] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.227172 7716 raft_consensus.cc:357] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.227771 7716 raft_consensus.cc:383] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:39.228030 7716 raft_consensus.cc:738] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.228207 7569 consensus_queue.cc:260] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.243561 7716 consensus_queue.cc:260] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.246575 7716 ts_tablet_manager.cc:1428] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20250626 19:58:39.252290 7568 tablet_bootstrap.cc:492] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:39.252722 7568 ts_tablet_manager.cc:1397] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.133s user 0.007s sys 0.007s
I20250626 19:58:39.261145 6712 catalog_manager.cc:5582] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130). New cstate: current_term: 1 leader_uuid: "315d304854c541bf8b5e6b1eea712d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:39.269023 7569 ts_tablet_manager.cc:1428] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20250626 19:58:39.271309 7568 raft_consensus.cc:357] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.271957 7568 raft_consensus.cc:383] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:39.272215 7568 raft_consensus.cc:738] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.272114 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9adef69fe7442a4836a84934e792c05" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
W20250626 19:58:39.273299 7294 leader_election.cc:343] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:39.274812 7569 tablet_bootstrap.cc:492] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:39.278398 7568 consensus_queue.cc:260] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.283030 7568 ts_tablet_manager.cc:1428] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20250626 19:58:39.283913 7568 tablet_bootstrap.cc:492] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:39.286690 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9adef69fe7442a4836a84934e792c05" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:39.287277 7216 raft_consensus.cc:3058] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.297344 7216 raft_consensus.cc:2466] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:39.298532 7293 leader_election.cc:304] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:39.299444 7867 raft_consensus.cc:2802] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:39.299855 7867 raft_consensus.cc:695] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:39.301288 7867 consensus_queue.cc:237] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.306560 7925 leader_election.cc:290] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:39.306298 6716 catalog_manager.cc:5582] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } }
I20250626 19:58:39.310078 7568 tablet_bootstrap.cc:654] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:39.325255 7568 tablet_bootstrap.cc:492] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:39.325659 7568 ts_tablet_manager.cc:1397] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.005s
I20250626 19:58:39.327966 7568 raft_consensus.cc:357] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.328593 7568 raft_consensus.cc:383] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:39.328848 7568 raft_consensus.cc:738] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.341864 7568 consensus_queue.cc:260] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.343670 7569 tablet_bootstrap.cc:654] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:39.348338 7568 ts_tablet_manager.cc:1428] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.022s user 0.003s sys 0.003s
I20250626 19:58:39.360177 7907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 19:58:39.368026 7568 tablet_bootstrap.cc:492] T 9de12a70dc90406c9ca8f58d253f35d5 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:39.381112 7907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: Bootstrap complete.
I20250626 19:58:39.405030 7867 raft_consensus.cc:491] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:39.405488 7867 raft_consensus.cc:513] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.407474 7867 leader_election.cc:290] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:39.409636 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83493d647cc8464e8e650bbca7c6072c" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:39.409672 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83493d647cc8464e8e650bbca7c6072c" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:39.410197 7071 raft_consensus.cc:2466] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:39.410197 7216 raft_consensus.cc:2466] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:39.411296 7294 leader_election.cc:304] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:39.411702 7568 tablet_bootstrap.cc:654] T 9de12a70dc90406c9ca8f58d253f35d5 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:39.412530 7867 raft_consensus.cc:2802] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:39.412902 7867 raft_consensus.cc:491] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:39.413264 7867 raft_consensus.cc:3058] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.414913 6716 catalog_manager.cc:5582] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } }
I20250626 19:58:39.417058 7569 tablet_bootstrap.cc:492] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:39.417474 7569 ts_tablet_manager.cc:1397] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.144s user 0.008s sys 0.005s
I20250626 19:58:39.419660 7867 raft_consensus.cc:513] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.419770 7569 raft_consensus.cc:357] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.420341 7569 raft_consensus.cc:383] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:39.420589 7569 raft_consensus.cc:738] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.421243 7569 consensus_queue.cc:260] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.425235 7569 ts_tablet_manager.cc:1428] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250626 19:58:39.431973 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83493d647cc8464e8e650bbca7c6072c" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:39.432492 7071 raft_consensus.cc:3058] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.439031 7071 raft_consensus.cc:2466] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:39.440068 7294 leader_election.cc:304] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:39.440878 7925 raft_consensus.cc:2802] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:39.441289 7925 raft_consensus.cc:695] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:39.442073 7925 consensus_queue.cc:237] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.450093 7569 tablet_bootstrap.cc:492] T 089e49e29b2b4f3dbea8a4597d9e36a2 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:39.450289 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83493d647cc8464e8e650bbca7c6072c" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:39.450789 7216 raft_consensus.cc:3058] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.450728 6712 catalog_manager.cc:5582] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130). New cstate: current_term: 1 leader_uuid: "315d304854c541bf8b5e6b1eea712d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:39.452486 6712 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 7b9a0f71535a4a65a775f728edbcc636 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
I20250626 19:58:39.458325 7216 raft_consensus.cc:2466] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:39.463867 7867 leader_election.cc:290] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:39.479977 7568 tablet_bootstrap.cc:492] T 9de12a70dc90406c9ca8f58d253f35d5 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:39.480454 7568 ts_tablet_manager.cc:1397] T 9de12a70dc90406c9ca8f58d253f35d5 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.113s user 0.008s sys 0.001s
I20250626 19:58:39.482715 7568 raft_consensus.cc:357] T 9de12a70dc90406c9ca8f58d253f35d5 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.484964 7568 raft_consensus.cc:383] T 9de12a70dc90406c9ca8f58d253f35d5 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:39.485203 7568 raft_consensus.cc:738] T 9de12a70dc90406c9ca8f58d253f35d5 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.486024 7569 tablet_bootstrap.cc:654] T 089e49e29b2b4f3dbea8a4597d9e36a2 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:39.485810 7568 consensus_queue.cc:260] T 9de12a70dc90406c9ca8f58d253f35d5 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.490520 7568 ts_tablet_manager.cc:1428] T 9de12a70dc90406c9ca8f58d253f35d5 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.010s user 0.001s sys 0.004s
I20250626 19:58:39.502154 7568 tablet_bootstrap.cc:492] T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:39.503432 7925 raft_consensus.cc:491] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:39.503880 7925 raft_consensus.cc:513] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.507314 7925 leader_election.cc:290] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:39.507488 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9de12a70dc90406c9ca8f58d253f35d5" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:39.508925 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9de12a70dc90406c9ca8f58d253f35d5" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:39.509476 7216 raft_consensus.cc:2466] T 9de12a70dc90406c9ca8f58d253f35d5 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:39.512748 7293 leader_election.cc:304] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
W20250626 19:58:39.513725 7294 leader_election.cc:343] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:39.515681 7925 raft_consensus.cc:2802] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:39.516070 7925 raft_consensus.cc:491] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:39.516500 7925 raft_consensus.cc:3058] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.520874 7569 tablet_bootstrap.cc:492] T 089e49e29b2b4f3dbea8a4597d9e36a2 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:39.521277 7569 ts_tablet_manager.cc:1397] T 089e49e29b2b4f3dbea8a4597d9e36a2 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.002s
I20250626 19:58:39.518535 7568 tablet_bootstrap.cc:654] T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:39.523587 7569 raft_consensus.cc:357] T 089e49e29b2b4f3dbea8a4597d9e36a2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.524199 7569 raft_consensus.cc:383] T 089e49e29b2b4f3dbea8a4597d9e36a2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:39.524467 7569 raft_consensus.cc:738] T 089e49e29b2b4f3dbea8a4597d9e36a2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.527005 7569 consensus_queue.cc:260] T 089e49e29b2b4f3dbea8a4597d9e36a2 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.545668 6716 catalog_manager.cc:5582] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } }
I20250626 19:58:39.548090 7569 ts_tablet_manager.cc:1428] T 089e49e29b2b4f3dbea8a4597d9e36a2 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.027s user 0.006s sys 0.001s
I20250626 19:58:39.548869 7932 raft_consensus.cc:491] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:39.549366 7932 raft_consensus.cc:513] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.551311 7932 leader_election.cc:290] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:39.551946 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "089e49e29b2b4f3dbea8a4597d9e36a2" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:39.552503 7071 raft_consensus.cc:2466] T 089e49e29b2b4f3dbea8a4597d9e36a2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
W20250626 19:58:39.553468 7152 leader_election.cc:343] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2
I20250626 19:58:39.554522 7150 leader_election.cc:304] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7
I20250626 19:58:39.555507 7932 raft_consensus.cc:2802] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:39.555876 7932 raft_consensus.cc:491] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:39.556300 7932 raft_consensus.cc:3058] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.558538 7867 raft_consensus.cc:491] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:39.559010 7867 raft_consensus.cc:513] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.560799 7867 leader_election.cc:290] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:39.562165 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b9a0f71535a4a65a775f728edbcc636" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:39.562716 7216 raft_consensus.cc:2466] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:39.563980 7293 leader_election.cc:304] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:39.565296 7867 raft_consensus.cc:2802] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:39.565733 7867 raft_consensus.cc:491] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:39.566048 7867 raft_consensus.cc:3058] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.566315 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b9a0f71535a4a65a775f728edbcc636" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:39.566936 7071 raft_consensus.cc:2466] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:39.571360 7569 tablet_bootstrap.cc:492] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:39.572975 7867 raft_consensus.cc:513] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.573551 6714 catalog_manager.cc:3460] renamed table default.table_76cdf24ffacc4d129b66c2fd6c0b93d8 [id=ad857025d66f4893872c8b5c92a9c0c7] in HMS: new name default.table_d04e336d056d484ab051754a2b012d9b
I20250626 19:58:39.577839 7569 tablet_bootstrap.cc:654] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:39.577311 6712 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 7b9a0f71535a4a65a775f728edbcc636 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
I20250626 19:58:39.579346 7932 raft_consensus.cc:513] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:39.583079 7152 leader_election.cc:343] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2
I20250626 19:58:39.583241 7568 tablet_bootstrap.cc:492] T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:39.583683 7568 ts_tablet_manager.cc:1397] T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.082s user 0.012s sys 0.000s
I20250626 19:58:39.583894 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "089e49e29b2b4f3dbea8a4597d9e36a2" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:39.584422 7071 raft_consensus.cc:3058] T 089e49e29b2b4f3dbea8a4597d9e36a2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.585490 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b9a0f71535a4a65a775f728edbcc636" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:39.585969 7216 raft_consensus.cc:3058] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.585857 7568 raft_consensus.cc:357] T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.587281 7568 raft_consensus.cc:383] T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:39.587535 7568 raft_consensus.cc:738] T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.587690 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b9a0f71535a4a65a775f728edbcc636" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:39.588212 7073 raft_consensus.cc:3058] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.588078 7568 consensus_queue.cc:260] T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.587965 7907 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:39.591307 7907 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae38ec946e14198ae4adc8dde9d66e8, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.592649 7216 raft_consensus.cc:2466] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:39.593713 7293 leader_election.cc:304] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:39.594343 7932 leader_election.cc:290] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:39.594846 7926 raft_consensus.cc:2802] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:39.542701 7925 raft_consensus.cc:513] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.601213 7568 ts_tablet_manager.cc:1428] T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20250626 19:58:39.603456 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9de12a70dc90406c9ca8f58d253f35d5" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:39.604013 7216 raft_consensus.cc:3058] T 9de12a70dc90406c9ca8f58d253f35d5 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.607164 7907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:39.608479 7072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9de12a70dc90406c9ca8f58d253f35d5" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
W20250626 19:58:39.609751 7294 leader_election.cc:343] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:39.610644 7925 leader_election.cc:290] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:39.612413 7867 leader_election.cc:290] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:39.616209 7867 raft_consensus.cc:491] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:39.616720 7867 raft_consensus.cc:513] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.621464 7216 raft_consensus.cc:2466] T 9de12a70dc90406c9ca8f58d253f35d5 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:39.625666 7072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7159554b45442b5aa5069d7a47d1f8e" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
W20250626 19:58:39.626979 7294 leader_election.cc:343] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:39.627565 6716 master_service.cc:948] ReplaceTablet: received request to replace tablet e95e6754e74a491087c4ea48e06c1e97 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:39.632673 7867 leader_election.cc:290] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:39.632440 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7159554b45442b5aa5069d7a47d1f8e" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:39.635996 7215 raft_consensus.cc:2466] T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:39.636345 7293 leader_election.cc:304] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:39.637416 7293 leader_election.cc:304] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:39.638398 7925 raft_consensus.cc:2802] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:39.638859 7925 raft_consensus.cc:695] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:39.639675 7925 consensus_queue.cc:237] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.642614 7867 raft_consensus.cc:2802] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:39.642989 7867 raft_consensus.cc:491] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:39.643339 7867 raft_consensus.cc:3058] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:39.643698 7953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:39.644469 7953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:58:39.646364 7569 tablet_bootstrap.cc:492] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:39.654234 7569 ts_tablet_manager.cc:1397] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.005s
I20250626 19:58:39.659934 7907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 19:58:39.669940 7569 raft_consensus.cc:357] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.671001 7569 raft_consensus.cc:383] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:39.672402 7569 raft_consensus.cc:738] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.639622 7073 raft_consensus.cc:2466] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:39.672441 7071 raft_consensus.cc:2466] T 089e49e29b2b4f3dbea8a4597d9e36a2 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:39.676116 7150 leader_election.cc:304] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7
I20250626 19:58:39.677070 7932 raft_consensus.cc:2802] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:39.677474 7932 raft_consensus.cc:695] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:39.678200 7932 consensus_queue.cc:237] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.678670 7407 heartbeater.cc:344] Connected to a master server at 127.4.46.189:36417
I20250626 19:58:39.685124 6716 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Sending DeleteTablet for 3 replicas of tablet e95e6754e74a491087c4ea48e06c1e97
I20250626 19:58:39.687134 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet e95e6754e74a491087c4ea48e06c1e97 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d677beae63a49ecbd11ed53e9e421fc) from {username='slave'} at 127.0.0.1:43274
I20250626 19:58:39.688575 7867 raft_consensus.cc:513] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.698117 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7159554b45442b5aa5069d7a47d1f8e" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:39.673933 7569 consensus_queue.cc:260] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:39.699332 7294 leader_election.cc:343] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:39.702626 6716 catalog_manager.cc:6652] ReplaceTablet: tablet e95e6754e74a491087c4ea48e06c1e97 deleted and replaced by tablet 9d677beae63a49ecbd11ed53e9e421fc
I20250626 19:58:39.703437 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet e95e6754e74a491087c4ea48e06c1e97 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d677beae63a49ecbd11ed53e9e421fc) from {username='slave'} at 127.0.0.1:54698
I20250626 19:58:39.704623 7569 ts_tablet_manager.cc:1428] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.050s user 0.005s sys 0.000s
I20250626 19:58:39.707111 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet e95e6754e74a491087c4ea48e06c1e97 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d677beae63a49ecbd11ed53e9e421fc) from {username='slave'} at 127.0.0.1:48028
I20250626 19:58:39.717329 7926 raft_consensus.cc:695] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
W20250626 19:58:39.722275 6716 catalog_manager.cc:5415] Ignoring report from unknown tablet 089e49e29b2b4f3dbea8a4597d9e36a2
I20250626 19:58:39.722323 7926 consensus_queue.cc:237] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.724431 6716 master_service.cc:948] ReplaceTablet: received request to replace tablet 734d9d3331b44fc49e1f64b8caf1e4d8 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:39.730710 7958 tablet_replica.cc:331] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:39.731570 7958 raft_consensus.cc:2241] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:39.735673 7569 tablet_bootstrap.cc:492] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:39.747393 7867 leader_election.cc:290] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:39.748409 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7159554b45442b5aa5069d7a47d1f8e" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:39.748951 7216 raft_consensus.cc:3058] T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:39.724067 6944 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9d677beae63a49ecbd11ed53e9e421fc has no consensus state
I20250626 19:58:39.752318 6715 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42822 (request call id 69) took 5251 ms. Trace:
I20250626 19:58:39.752676 6715 rpcz_store.cc:276] 0626 19:58:34.500844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:34.500970 (+ 126us) service_pool.cc:225] Handling call
0626 19:58:34.534856 (+ 33886us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b54000203d4. stack: 00007efd765d2483 00007efd765d2380 00007efd7614e574 00000000004fa174 00000000004fa109 00007efd7d4d4a70 00007efd7d558b1a 00007efd7f205fac 00007efd7f300046 00007efd7d0211a4 00007efd7d021132 00007efd7d021061 00007efd7d020fdc 00007efd7d020292 00007efd7b318dcc 00007efd7b318206
0626 19:58:36.271524 (+1736668us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:39.752253 (+3480729us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":50884,"Hive Metastore.run_cpu_time_us":348,"Hive Metastore.run_wall_time_us":515521,"spinlock_wait_cycles":39923456}
I20250626 19:58:39.755358 6718 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42822 (request call id 82) took 4207 ms. Trace:
I20250626 19:58:39.754824 7958 raft_consensus.cc:2270] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 19:58:39.755949 6711 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_dc0e9650e86e4c60aec4f82f9b82f959 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:58:39.758287 6713 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6f04f2a6bf414cb8912fe28aff625d2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:39.758734 7958 ts_tablet_manager.cc:1905] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:58:39.759863 6709 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_acc449d45fc742dab0b039b077cc75e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:39.755707 6718 rpcz_store.cc:276] 0626 19:58:35.547578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:35.598688 (+ 51110us) service_pool.cc:225] Handling call
0626 19:58:36.394837 (+796149us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:39.755295 (+3360458us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":1261588,"Hive Metastore.run_cpu_time_us":311,"Hive Metastore.run_wall_time_us":433218}
I20250626 19:58:39.765265 7960 tablet_replica.cc:331] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:39.766413 6710 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42822 (request call id 70) took 5265 ms. Trace:
I20250626 19:58:39.769572 7960 raft_consensus.cc:2241] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:39.770241 7960 raft_consensus.cc:2270] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 19:58:39.771741 6717 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_63a5ae538eef4f739a22697dfcfb56c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:39.782578 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 9d677beae63a49ecbd11ed53e9e421fc (DEFAULT_TABLE table=default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:39.783653 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d677beae63a49ecbd11ed53e9e421fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:39.784562 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 9d677beae63a49ecbd11ed53e9e421fc (DEFAULT_TABLE table=default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:39.785521 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d677beae63a49ecbd11ed53e9e421fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:39.784721 6710 rpcz_store.cc:276] 0626 19:58:34.501187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:34.501974 (+ 787us) service_pool.cc:225] Handling call
0626 19:58:34.537100 (+ 35126us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b54000203d4. stack: 00007efd765d2483 00007efd765d2380 00007efd7614e574 00000000004fa174 00000000004fa109 00007efd7d4d4a70 00007efd7d558b1a 00007efd7f205fac 00007efd7f3003b6 00007efd7d025ca4 00007efd7d025c32 00007efd7d025b61 00007efd7d025adc 00007efd7d024d92 00007efd7b318dcc 00007efd7b318206
0626 19:58:36.272364 (+1735264us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:39.757115 (+3484751us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:39.765443 (+ 8328us) catalog_manager.cc:3683] Apply alter schema
0626 19:58:39.766350 (+ 907us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":565487,"Hive Metastore.run_cpu_time_us":1831,"Hive Metastore.run_wall_time_us":452903,"spinlock_wait_cycles":37916928}
I20250626 19:58:39.787547 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 9d677beae63a49ecbd11ed53e9e421fc (DEFAULT_TABLE table=default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:39.793401 7216 raft_consensus.cc:2466] T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:39.800206 7959 tablet_replica.cc:331] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:39.801174 7293 leader_election.cc:304] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:39.802279 7867 raft_consensus.cc:2802] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:39.802523 7960 ts_tablet_manager.cc:1905] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:39.802794 7867 raft_consensus.cc:695] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:39.803324 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d677beae63a49ecbd11ed53e9e421fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:39.803560 7867 consensus_queue.cc:237] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.806524 7569 tablet_bootstrap.cc:654] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:39.812973 6712 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
I20250626 19:58:39.807346 7959 raft_consensus.cc:2241] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
W20250626 19:58:39.813387 6712 catalog_manager.cc:5415] Ignoring report from unknown tablet 089e49e29b2b4f3dbea8a4597d9e36a2
I20250626 19:58:39.815702 7959 raft_consensus.cc:2270] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:39.833659 7959 ts_tablet_manager.cc:1905] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:39.849956 7569 tablet_bootstrap.cc:492] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:39.850498 7569 ts_tablet_manager.cc:1397] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.115s user 0.004s sys 0.006s
I20250626 19:58:39.864777 6716 catalog_manager.cc:5582] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } }
I20250626 19:58:39.861016 7569 raft_consensus.cc:357] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.866472 7569 raft_consensus.cc:383] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:39.866820 7569 raft_consensus.cc:738] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:39.867820 7926 consensus_queue.cc:1035] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:39.868604 7569 consensus_queue.cc:260] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.883621 7958 ts_tablet_manager.cc:1918] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:39.884009 7958 log.cc:1199] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/e95e6754e74a491087c4ea48e06c1e97
I20250626 19:58:39.885037 7119 heartbeater.cc:344] Connected to a master server at 127.4.46.189:36417
I20250626 19:58:39.895248 6716 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 7b9a0f71535a4a65a775f728edbcc636 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
I20250626 19:58:39.896823 7972 tablet_bootstrap.cc:492] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:39.898224 6716 catalog_manager.cc:5582] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } }
I20250626 19:58:39.903213 7568 tablet_bootstrap.cc:492] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:39.905421 7958 ts_tablet_manager.cc:1939] T e95e6754e74a491087c4ea48e06c1e97 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:39.908242 6705 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet e95e6754e74a491087c4ea48e06c1e97 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]) successfully deleted
I20250626 19:58:39.908962 7568 tablet_bootstrap.cc:654] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:39.934098 7932 consensus_queue.cc:1035] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:39.942297 7569 ts_tablet_manager.cc:1428] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.087s user 0.000s sys 0.004s
I20250626 19:58:39.943979 6712 master_service.cc:948] ReplaceTablet: received request to replace tablet 734d9d3331b44fc49e1f64b8caf1e4d8 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:39.947003 7568 tablet_bootstrap.cc:492] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:39.947444 7568 ts_tablet_manager.cc:1397] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.000s
I20250626 19:58:39.959323 6712 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_daf65ec234534a069c2d1ad34f13952a" } modify_external_catalogs: true
I20250626 19:58:39.960376 6712 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_daf65ec234534a069c2d1ad34f13952a" } modify_external_catalogs: true
I20250626 19:58:39.964360 6718 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_5283819977d9487a9d1aaae326249a46" } modify_external_catalogs: true
I20250626 19:58:39.965030 6718 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_5283819977d9487a9d1aaae326249a46" } modify_external_catalogs: true
I20250626 19:58:39.955073 7742 raft_consensus.cc:491] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:39.979151 7742 raft_consensus.cc:513] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.983870 6710 master_service.cc:948] ReplaceTablet: received request to replace tablet 734d9d3331b44fc49e1f64b8caf1e4d8 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:39.984005 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9adef69fe7442a4836a84934e792c05" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:39.986999 7215 raft_consensus.cc:2391] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbe151be01e24c82854d9c8c79c797ca in current term 1: Already voted for candidate 297d2c913667408bae82668a557e06c7 in this term.
I20250626 19:58:39.989971 6711 catalog_manager.cc:2149] created HMS catalog entry for table [id=d1d807f596fd4519b4d74850a971205a]
I20250626 19:58:39.991377 7568 raft_consensus.cc:357] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:39.992323 6710 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_6a693c4bae214626a4050c7a94276f98" } new_table_name: "default.Table_30a8077ecd80407991c244d577d16cad" modify_external_catalogs: true
I20250626 19:58:39.999085 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9adef69fe7442a4836a84934e792c05" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:39.981119 7742 leader_election.cc:290] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:40.015969 7568 raft_consensus.cc:383] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:40.016364 7568 raft_consensus.cc:738] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:40.017093 7568 consensus_queue.cc:260] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:40.062507 7009 leader_election.cc:304] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:40.065904 7932 consensus_queue.cc:1035] T fabf0275e32e400c9731eb09d9c2620c P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:40.073458 7742 raft_consensus.cc:3058] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:40.081768 7742 raft_consensus.cc:2747] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:40.089396 6711 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 105) took 1529 ms. Trace:
I20250626 19:58:40.089733 6711 rpcz_store.cc:276] 0626 19:58:38.559724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:38.716180 (+156456us) service_pool.cc:225] Handling call
0626 19:58:39.756356 (+1040176us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:39.759618 (+ 3262us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:39.989929 (+230311us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:40.007263 (+ 17334us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:40.089045 (+ 81782us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:40.089323 (+ 278us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:39.991614 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:39.991990 (+ 376us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:39.992014 (+ 24us) write_op.cc:435] Acquired schema lock
0626 19:58:39.992025 (+ 11us) tablet.cc:660] Decoding operations
0626 19:58:39.992296 (+ 271us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:39.992334 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:39.992358 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:39.992539 (+ 181us) tablet.cc:699] Row locks acquired
0626 19:58:39.992551 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:58:39.992652 (+ 101us) write_op.cc:270] Start()
0626 19:58:39.992731 (+ 79us) write_op.cc:276] Timestamp: P: 1750967919992685 usec, L: 0
0626 19:58:39.992748 (+ 17us) op_driver.cc:348] REPLICATION: starting
0626 19:58:39.993224 (+ 476us) log.cc:844] Serialized 433 byte log entry
0626 19:58:40.003073 (+ 9849us) op_driver.cc:464] REPLICATION: finished
0626 19:58:40.003432 (+ 359us) write_op.cc:301] APPLY: starting
0626 19:58:40.003496 (+ 64us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:40.003598 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:40.003682 (+ 84us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:40.004024 (+ 342us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:40.004148 (+ 124us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:40.004207 (+ 59us) write_op.cc:312] APPLY: finished
0626 19:58:40.004378 (+ 171us) log.cc:844] Serialized 29 byte log entry
0626 19:58:40.004510 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:40.004575 (+ 65us) write_op.cc:454] Released schema lock
0626 19:58:40.004610 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0626 19:58:40.004820 (+ 210us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:40.004843 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":2099,"Hive Metastore.run_cpu_time_us":652,"Hive Metastore.run_wall_time_us":227342,"child_traces":[["op",{"apply.queue_time_us":214,"apply.run_cpu_time_us":1354,"apply.run_wall_time_us":1353,"num_ops":2,"prepare.queue_time_us":215,"prepare.run_cpu_time_us":1966,"prepare.run_wall_time_us":4440,"raft.queue_time_us":89,"raft.run_cpu_time_us":558,"raft.run_wall_time_us":559,"replication_time_us":10236}]]}
W20250626 19:58:40.096077 6703 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9d677beae63a49ecbd11ed53e9e421fc consensus state has no leader
I20250626 19:58:40.096690 6714 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42822 (request call id 91) took 3491 ms. Trace:
I20250626 19:58:40.096979 6714 rpcz_store.cc:276] 0626 19:58:36.605572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:36.605688 (+ 116us) service_pool.cc:225] Handling call
0626 19:58:38.941342 (+2335654us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:40.035571 (+1094229us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:40.095250 (+ 59679us) catalog_manager.cc:3683] Apply alter schema
0626 19:58:40.096631 (+ 1381us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":85,"Hive Metastore.run_cpu_time_us":1744,"Hive Metastore.run_wall_time_us":631047}
I20250626 19:58:40.102666 6716 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:40.105028 7568 ts_tablet_manager.cc:1428] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.157s user 0.007s sys 0.000s
I20250626 19:58:40.091874 7980 raft_consensus.cc:491] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:40.107882 7980 raft_consensus.cc:513] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:40.109760 7980 leader_election.cc:290] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:40.111326 7569 tablet_bootstrap.cc:492] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:40.112301 7972 tablet_bootstrap.cc:700] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7: Time spent opening tablet: real 0.190s user 0.002s sys 0.000s
I20250626 19:58:40.111308 7216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7159554b45442b5aa5069d7a47d1f8e" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:40.113965 7216 raft_consensus.cc:2391] T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbe151be01e24c82854d9c8c79c797ca in current term 1: Already voted for candidate 297d2c913667408bae82668a557e06c7 in this term.
I20250626 19:58:40.121601 7569 tablet_bootstrap.cc:654] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:40.142282 7985 log.cc:927] Time spent T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: Append to log took a long time: real 0.064s user 0.006s sys 0.002s
I20250626 19:58:40.143633 7926 consensus_queue.cc:1035] T 55fad5c8639047a0a5519243d6ae76a3 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:40.162276 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7159554b45442b5aa5069d7a47d1f8e" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:40.166281 7932 consensus_queue.cc:1035] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:40.167797 7972 tablet_bootstrap.cc:654] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:40.180701 7960 ts_tablet_manager.cc:1918] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:40.181082 7960 log.cc:1199] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/e95e6754e74a491087c4ea48e06c1e97
I20250626 19:58:40.181809 7960 ts_tablet_manager.cc:1939] T e95e6754e74a491087c4ea48e06c1e97 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:40.184684 7972 tablet_bootstrap.cc:492] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:40.184101 6714 catalog_manager.cc:5582] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } }
I20250626 19:58:40.194331 7972 ts_tablet_manager.cc:1397] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.298s user 0.012s sys 0.000s
I20250626 19:58:40.201894 6715 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:40.222558 7994 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:40.224895 6713 catalog_manager.cc:2149] created HMS catalog entry for table [id=912c0fafb9264e07b1fa485ea0d193d9]
I20250626 19:58:40.229166 7216 raft_consensus.cc:1273] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Refusing update from remote peer 297d2c913667408bae82668a557e06c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:58:40.230734 7867 consensus_queue.cc:1035] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:40.239521 7009 leader_election.cc:304] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:40.241498 6713 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 98) took 2596 ms. Trace:
I20250626 19:58:40.241879 6713 rpcz_store.cc:276] 0626 19:58:37.645521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:37.646469 (+ 948us) service_pool.cc:225] Handling call
0626 19:58:39.758703 (+2112234us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:39.762733 (+ 4030us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:40.224851 (+462118us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:40.240815 (+ 15964us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:40.241145 (+ 330us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:40.241439 (+ 294us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:40.226524 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:40.226871 (+ 347us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:40.226892 (+ 21us) write_op.cc:435] Acquired schema lock
0626 19:58:40.226904 (+ 12us) tablet.cc:660] Decoding operations
0626 19:58:40.227144 (+ 240us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:40.227175 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:40.227198 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:40.227371 (+ 173us) tablet.cc:699] Row locks acquired
0626 19:58:40.227385 (+ 14us) write_op.cc:260] PREPARE: finished
0626 19:58:40.227477 (+ 92us) write_op.cc:270] Start()
0626 19:58:40.227547 (+ 70us) write_op.cc:276] Timestamp: P: 1750967920227463 usec, L: 0
0626 19:58:40.227562 (+ 15us) op_driver.cc:348] REPLICATION: starting
0626 19:58:40.227959 (+ 397us) log.cc:844] Serialized 433 byte log entry
0626 19:58:40.236387 (+ 8428us) op_driver.cc:464] REPLICATION: finished
0626 19:58:40.237892 (+ 1505us) write_op.cc:301] APPLY: starting
0626 19:58:40.237934 (+ 42us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:40.238023 (+ 89us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:40.238102 (+ 79us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:40.238445 (+ 343us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:40.238566 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:40.238609 (+ 43us) write_op.cc:312] APPLY: finished
0626 19:58:40.238768 (+ 159us) log.cc:844] Serialized 29 byte log entry
0626 19:58:40.238908 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:40.238959 (+ 51us) write_op.cc:454] Released schema lock
0626 19:58:40.238988 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":242017,"Hive Metastore.run_cpu_time_us":590,"Hive Metastore.run_wall_time_us":218860,"child_traces":[["op",{"apply.queue_time_us":1397,"apply.run_cpu_time_us":1241,"apply.run_wall_time_us":1240,"num_ops":2,"prepare.queue_time_us":125,"raft.queue_time_us":1707,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":10,"replication_time_us":8750,"threads_started":1}]]}
I20250626 19:58:40.261525 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 196180f1e78a4d5ca8aca8d7bb666d42 (DEFAULT_TABLE table=default.table_dc0e9650e86e4c60aec4f82f9b82f959 [id=d1d807f596fd4519b4d74850a971205a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:40.279407 6713 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 7b9a0f71535a4a65a775f728edbcc636 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
I20250626 19:58:40.280771 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 196180f1e78a4d5ca8aca8d7bb666d42 (DEFAULT_TABLE table=default.table_dc0e9650e86e4c60aec4f82f9b82f959 [id=d1d807f596fd4519b4d74850a971205a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:40.281706 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 196180f1e78a4d5ca8aca8d7bb666d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:40.284008 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 196180f1e78a4d5ca8aca8d7bb666d42 (DEFAULT_TABLE table=default.table_dc0e9650e86e4c60aec4f82f9b82f959 [id=d1d807f596fd4519b4d74850a971205a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:40.284973 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 196180f1e78a4d5ca8aca8d7bb666d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:40.226400 7953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 188d02ec81e645229acdc5eed98c9f9f. Latest consensus state: current_term: 1 leader_uuid: "188d02ec81e645229acdc5eed98c9f9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:40.287866 7953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:58:40.288491 7978 mvcc.cc:204] Tried to move back new op lower bound from 7171964601240424448 to 7171964599216623616. Current Snapshot: MvccSnapshot[applied={T|T < 7171964601240424448}]
I20250626 19:58:40.294639 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 196180f1e78a4d5ca8aca8d7bb666d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:40.305305 6714 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42822:
table { table_name: "default.Table_dc0e9650e86e4c60aec4f82f9b82f959" } new_table_name: "default.Table_fd11e459e1d9471fae895dcdf01e89fa" modify_external_catalogs: true
I20250626 19:58:40.307039 6715 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_09c2ccbfee7e48e4a354a5138f0855d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:40.311719 7814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39222 (request call id 0) took 1408 ms. Trace:
I20250626 19:58:40.326100 7994 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:40.331727 7337 tablet_service.cc:1468] Processing CreateTablet for tablet 0487e1d34dd74d70a5b5c8ac8cc0f636 (DEFAULT_TABLE table=default.table_6f04f2a6bf414cb8912fe28aff625d2e [id=912c0fafb9264e07b1fa485ea0d193d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:40.332901 7337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0487e1d34dd74d70a5b5c8ac8cc0f636. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:40.333901 7193 tablet_service.cc:1468] Processing CreateTablet for tablet 0487e1d34dd74d70a5b5c8ac8cc0f636 (DEFAULT_TABLE table=default.table_6f04f2a6bf414cb8912fe28aff625d2e [id=912c0fafb9264e07b1fa485ea0d193d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:40.335187 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0487e1d34dd74d70a5b5c8ac8cc0f636. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:40.341601 7867 consensus_queue.cc:1035] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:40.349989 6702 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet e95e6754e74a491087c4ea48e06c1e97 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]) successfully deleted
I20250626 19:58:40.351251 7072 raft_consensus.cc:1273] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Refusing update from remote peer 297d2c913667408bae82668a557e06c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:58:40.356276 7932 consensus_queue.cc:1035] T 40b6570e38434e0c8301d010e96c67a3 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:40.358444 7998 tablet.cc:1722] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:58:40.363370 7999 tablet.cc:1722] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:58:40.393594 7568 tablet_bootstrap.cc:492] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:40.393635 6709 catalog_manager.cc:2149] created HMS catalog entry for table [id=11ee0dd5a39742399865b36ea597920d]
I20250626 19:58:40.407244 6709 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 101) took 2503 ms. Trace:
I20250626 19:58:40.409454 6713 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 7b9a0f71535a4a65a775f728edbcc636 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
I20250626 19:58:40.407577 6709 rpcz_store.cc:276] 0626 19:58:37.903487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:37.908370 (+ 4883us) service_pool.cc:225] Handling call
0626 19:58:39.760203 (+1851833us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:39.767964 (+ 7761us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:40.393594 (+625630us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:40.406623 (+ 13029us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:40.406914 (+ 291us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:40.407186 (+ 272us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:40.395202 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:40.395568 (+ 366us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:40.395591 (+ 23us) write_op.cc:435] Acquired schema lock
0626 19:58:40.395604 (+ 13us) tablet.cc:660] Decoding operations
0626 19:58:40.395887 (+ 283us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:40.395930 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:40.395950 (+ 20us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:40.396156 (+ 206us) tablet.cc:699] Row locks acquired
0626 19:58:40.396168 (+ 12us) write_op.cc:260] PREPARE: finished
0626 19:58:40.396279 (+ 111us) write_op.cc:270] Start()
0626 19:58:40.396355 (+ 76us) write_op.cc:276] Timestamp: P: 1750967920396291 usec, L: 0
0626 19:58:40.396371 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 19:58:40.396810 (+ 439us) log.cc:844] Serialized 433 byte log entry
0626 19:58:40.404101 (+ 7291us) op_driver.cc:464] REPLICATION: finished
0626 19:58:40.404405 (+ 304us) write_op.cc:301] APPLY: starting
0626 19:58:40.404453 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:40.404549 (+ 96us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:40.404629 (+ 80us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:40.404987 (+ 358us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:40.405116 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:40.405159 (+ 43us) write_op.cc:312] APPLY: finished
0626 19:58:40.405314 (+ 155us) log.cc:844] Serialized 29 byte log entry
0626 19:58:40.405455 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:40.405516 (+ 61us) write_op.cc:454] Released schema lock
0626 19:58:40.405550 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0626 19:58:40.405754 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:40.405777 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":454997,"Hive Metastore.run_cpu_time_us":589,"Hive Metastore.run_wall_time_us":164010,"child_traces":[["op",{"apply.queue_time_us":174,"apply.run_cpu_time_us":1321,"apply.run_wall_time_us":1320,"num_ops":2,"prepare.queue_time_us":154,"prepare.run_cpu_time_us":1891,"prepare.run_wall_time_us":1891,"raft.queue_time_us":1241,"raft.run_cpu_time_us":656,"raft.run_wall_time_us":656,"replication_time_us":7649}]]}
I20250626 19:58:40.420445 7867 consensus_queue.cc:1035] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:40.434772 7568 tablet_bootstrap.cc:654] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:40.435207 7053 tablet_service.cc:1468] Processing CreateTablet for tablet 0487e1d34dd74d70a5b5c8ac8cc0f636 (DEFAULT_TABLE table=default.table_6f04f2a6bf414cb8912fe28aff625d2e [id=912c0fafb9264e07b1fa485ea0d193d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:40.436311 7053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0487e1d34dd74d70a5b5c8ac8cc0f636. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:40.283247 7980 raft_consensus.cc:3058] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:40.445598 7867 consensus_queue.cc:1035] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:40.460402 6709 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_afaa8d56134e47008a0eb5b4de5c35da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:40.456229 7959 ts_tablet_manager.cc:1918] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:40.464757 7959 log.cc:1199] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/e95e6754e74a491087c4ea48e06c1e97
I20250626 19:58:40.469475 6717 catalog_manager.cc:2149] created HMS catalog entry for table [id=22351693a59e46a9b9491eb79f1d0fe0]
I20250626 19:58:40.468874 7952 consensus_queue.cc:1035] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250626 19:58:40.474886 7972 raft_consensus.cc:357] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:40.478416 7972 raft_consensus.cc:383] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:40.478739 7972 raft_consensus.cc:738] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:40.479729 7972 consensus_queue.cc:260] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:40.498739 6717 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 99) took 2785 ms. Trace:
I20250626 19:58:40.498847 7980 raft_consensus.cc:2747] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:40.499209 6717 rpcz_store.cc:276] 0626 19:58:37.713651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:37.713793 (+ 142us) service_pool.cc:225] Handling call
0626 19:58:39.771547 (+2057754us) spinlock_profiling.cc:243] Waited 4.53 ms on lock 0x7b4000001878. stack: 00007efd765d2483 00007efd765d2380 00007efd7614e574 00000000004fa174 00000000004fa109 00007efd764479a1 00007efd765c71dc 00007efd765c6d9b 00007efd7f23b8f9 00007efd7f23b890 00007efd7f37c3f4 00007efd7f2259d1 00007efd7f1dd842 00007efd7f1e156e 00007efd7f2ffd6f 00007efd7d01c6a4
0626 19:58:39.773462 (+ 1915us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:39.776349 (+ 2887us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:40.469433 (+693084us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:40.494477 (+ 25044us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 19:58:40.498255 (+ 3778us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 19:58:40.498656 (+ 401us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:40.471442 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:40.471862 (+ 420us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:40.471884 (+ 22us) write_op.cc:435] Acquired schema lock
0626 19:58:40.471898 (+ 14us) tablet.cc:660] Decoding operations
0626 19:58:40.472212 (+ 314us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:40.472263 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:40.472292 (+ 29us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:40.472554 (+ 262us) tablet.cc:699] Row locks acquired
0626 19:58:40.472570 (+ 16us) write_op.cc:260] PREPARE: finished
0626 19:58:40.472686 (+ 116us) write_op.cc:270] Start()
0626 19:58:40.472779 (+ 93us) write_op.cc:276] Timestamp: P: 1750967920472707 usec, L: 0
0626 19:58:40.472799 (+ 20us) op_driver.cc:348] REPLICATION: starting
0626 19:58:40.473337 (+ 538us) log.cc:844] Serialized 433 byte log entry
0626 19:58:40.481098 (+ 7761us) op_driver.cc:464] REPLICATION: finished
0626 19:58:40.481414 (+ 316us) write_op.cc:301] APPLY: starting
0626 19:58:40.481470 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:40.481582 (+ 112us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:40.481668 (+ 86us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:40.482072 (+ 404us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:40.482223 (+ 151us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:40.482263 (+ 40us) write_op.cc:312] APPLY: finished
0626 19:58:40.482445 (+ 182us) log.cc:844] Serialized 29 byte log entry
0626 19:58:40.482620 (+ 175us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:40.482687 (+ 67us) write_op.cc:454] Released schema lock
0626 19:58:40.482723 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0626 19:58:40.482973 (+ 250us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:40.482996 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":611596,"Hive Metastore.run_cpu_time_us":600,"Hive Metastore.run_wall_time_us":80658,"spinlock_wait_cycles":10416384,"child_traces":[["op",{"apply.queue_time_us":183,"apply.run_cpu_time_us":1521,"apply.run_wall_time_us":1521,"num_ops":2,"prepare.queue_time_us":124,"prepare.run_cpu_time_us":2205,"prepare.run_wall_time_us":2204,"raft.queue_time_us":1142,"raft.run_cpu_time_us":653,"raft.run_wall_time_us":652,"replication_time_us":8212}]]}
I20250626 19:58:40.526497 7193 tablet_service.cc:1468] Processing CreateTablet for tablet 2ced063ac30b4168b4af4cf2ba239e25 (DEFAULT_TABLE table=default.table_acc449d45fc742dab0b039b077cc75e0 [id=11ee0dd5a39742399865b36ea597920d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:40.542461 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ced063ac30b4168b4af4cf2ba239e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:40.525276 7337 tablet_service.cc:1468] Processing CreateTablet for tablet 2ced063ac30b4168b4af4cf2ba239e25 (DEFAULT_TABLE table=default.table_acc449d45fc742dab0b039b077cc75e0 [id=11ee0dd5a39742399865b36ea597920d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:40.525017 7952 consensus_queue.cc:1035] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:40.550722 7337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ced063ac30b4168b4af4cf2ba239e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:40.556677 7340 tablet_service.cc:1468] Processing CreateTablet for tablet a5646bc4c82d4c23941b27994648ee2e (DEFAULT_TABLE table=default.table_63a5ae538eef4f739a22697dfcfb56c7 [id=22351693a59e46a9b9491eb79f1d0fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:40.557811 7340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5646bc4c82d4c23941b27994648ee2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:40.577023 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 2ced063ac30b4168b4af4cf2ba239e25 (DEFAULT_TABLE table=default.table_acc449d45fc742dab0b039b077cc75e0 [id=11ee0dd5a39742399865b36ea597920d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:40.544672 7262 heartbeater.cc:344] Connected to a master server at 127.4.46.189:36417
W20250626 19:58:40.569893 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:58:40.576447 7194 tablet_service.cc:1468] Processing CreateTablet for tablet a5646bc4c82d4c23941b27994648ee2e (DEFAULT_TABLE table=default.table_63a5ae538eef4f739a22697dfcfb56c7 [id=22351693a59e46a9b9491eb79f1d0fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:40.580165 7568 tablet_bootstrap.cc:492] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:40.590209 7568 ts_tablet_manager.cc:1397] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.197s user 0.010s sys 0.000s
I20250626 19:58:40.590842 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5646bc4c82d4c23941b27994648ee2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:40.592305 7568 raft_consensus.cc:357] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:40.592885 7568 raft_consensus.cc:383] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:40.593127 7568 raft_consensus.cc:738] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:40.593647 7568 consensus_queue.cc:260] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:40.596877 7568 ts_tablet_manager.cc:1428] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250626 19:58:40.601008 7568 tablet_bootstrap.cc:492] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:40.605996 7568 tablet_bootstrap.cc:654] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
W20250626 19:58:40.613648 6713 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
I20250626 19:58:40.614005 6711 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_df2c3f4a394c42cc81d626c1e46157f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:40.640345 7568 tablet_bootstrap.cc:492] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:40.640751 7568 ts_tablet_manager.cc:1397] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.002s
I20250626 19:58:40.652822 7972 ts_tablet_manager.cc:1428] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.455s user 0.005s sys 0.000s
I20250626 19:58:40.653827 7972 tablet_bootstrap.cc:492] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:40.665416 6716 master_service.cc:948] ReplaceTablet: received request to replace tablet e95e6754e74a491087c4ea48e06c1e97 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:40.666743 7568 raft_consensus.cc:357] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:40.667412 7568 raft_consensus.cc:383] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:40.667661 7568 raft_consensus.cc:738] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:40.668535 7568 consensus_queue.cc:260] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:40.681340 7972 tablet_bootstrap.cc:654] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:40.709767 8011 raft_consensus.cc:491] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:40.710291 8011 raft_consensus.cc:513] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:40.714910 8011 leader_election.cc:290] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:40.714463 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "196180f1e78a4d5ca8aca8d7bb666d42" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
W20250626 19:58:40.723311 7152 leader_election.cc:343] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:40.728879 7867 consensus_queue.cc:1035] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:40.732834 7932 consensus_queue.cc:1035] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250626 19:58:40.745730 7787 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" instance_seqno: 1750967903336488) as {username='slave'} at 127.4.46.130:58375; Asking this server to re-register.
I20250626 19:58:40.748011 7568 ts_tablet_manager.cc:1428] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.107s user 0.007s sys 0.000s
I20250626 19:58:40.752753 6713 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49] alter complete (version 1)
I20250626 19:58:40.588057 7051 tablet_service.cc:1468] Processing CreateTablet for tablet a5646bc4c82d4c23941b27994648ee2e (DEFAULT_TABLE table=default.table_63a5ae538eef4f739a22697dfcfb56c7 [id=22351693a59e46a9b9491eb79f1d0fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:40.732987 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "196180f1e78a4d5ca8aca8d7bb666d42" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
W20250626 19:58:40.759843 7294 consensus_peers.cc:487] T 9de12a70dc90406c9ca8f58d253f35d5 P 297d2c913667408bae82668a557e06c7 -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250626 19:58:40.759989 7150 leader_election.cc:343] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:40.754113 7262 heartbeater.cc:461] Registering TS with master...
I20250626 19:58:40.760394 7150 leader_election.cc:304] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a; no voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:40.762795 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5646bc4c82d4c23941b27994648ee2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:40.761514 8011 raft_consensus.cc:2747] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:40.770618 7568 tablet_bootstrap.cc:492] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:40.773252 7787 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "297d2c913667408bae82668a557e06c7" instance_seqno: 1750967905242709) as {username='slave'} at 127.4.46.131:44243; Asking this server to re-register.
I20250626 19:58:40.773008 6716 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "default.Table_1d890136b93b48cab4d218cbd6a72bba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:40.788295 7407 heartbeater.cc:461] Registering TS with master...
I20250626 19:58:40.789928 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ced063ac30b4168b4af4cf2ba239e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:40.792200 7972 tablet_bootstrap.cc:492] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:40.798902 7972 ts_tablet_manager.cc:1397] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.145s user 0.007s sys 0.005s
I20250626 19:58:40.803174 7959 ts_tablet_manager.cc:1939] T e95e6754e74a491087c4ea48e06c1e97 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:40.814903 7959 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54698 (request call id 133) took 1111 ms. Trace:
I20250626 19:58:40.812461 7569 tablet_bootstrap.cc:492] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:40.815912 7867 consensus_queue.cc:1035] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:40.816498 7569 ts_tablet_manager.cc:1397] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.705s user 0.014s sys 0.007s
I20250626 19:58:40.820817 7994 leader_election.cc:290] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
I20250626 19:58:40.802042 7972 raft_consensus.cc:357] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:40.824393 7972 raft_consensus.cc:383] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:40.824977 7787 ts_manager.cc:194] Registered new tserver with Master: 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:40.830020 7972 raft_consensus.cc:738] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:40.815392 7959 rpcz_store.cc:276] 0626 19:58:39.703109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:39.703265 (+ 156us) service_pool.cc:225] Handling call
0626 19:58:39.729925 (+ 26660us) ts_tablet_manager.cc:1145] Deleting tablet e95e6754e74a491087c4ea48e06c1e97
0626 19:58:39.800066 (+ 70141us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:58:40.106029 (+305963us) tablet_metadata.cc:641] Metadata flushed
0626 19:58:40.455961 (+349932us) tablet_metadata.cc:641] Metadata flushed
0626 19:58:40.814842 (+358881us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":25232,"thread_start_us":24141,"threads_started":1}
I20250626 19:58:40.834501 7952 consensus_queue.cc:1035] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:40.834908 6703 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet e95e6754e74a491087c4ea48e06c1e97 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]) successfully deleted
I20250626 19:58:40.835279 7972 consensus_queue.cc:260] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:40.838922 7569 raft_consensus.cc:357] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:40.839700 7569 raft_consensus.cc:383] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:40.839954 7569 raft_consensus.cc:738] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:40.841601 7972 ts_tablet_manager.cc:1428] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20250626 19:58:40.841624 7569 consensus_queue.cc:260] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:40.853835 7786 ts_manager.cc:194] Registered new tserver with Master: 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:40.855224 7568 tablet_bootstrap.cc:654] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:40.857558 8018 raft_consensus.cc:491] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:40.858070 8018 raft_consensus.cc:513] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:40.861101 8018 leader_election.cc:290] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:40.861312 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d677beae63a49ecbd11ed53e9e421fc" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:40.864020 7215 raft_consensus.cc:2466] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:40.865839 7071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d677beae63a49ecbd11ed53e9e421fc" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:40.866438 7569 ts_tablet_manager.cc:1428] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.050s user 0.006s sys 0.001s
I20250626 19:58:40.867322 7569 tablet_bootstrap.cc:492] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
W20250626 19:58:40.867309 7294 leader_election.cc:343] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:40.874518 6879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 162 } } ignore_live_leader: false dest_uuid: "6610553ba0df459085f7373935774377" is_pre_election: true
I20250626 19:58:40.878794 7293 leader_election.cc:304] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:40.886643 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 162 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:40.888618 7756 leader_election.cc:304] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ae38ec946e14198ae4adc8dde9d66e8; no voters: 188d02ec81e645229acdc5eed98c9f9f, 6610553ba0df459085f7373935774377
I20250626 19:58:40.890091 7994 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:58:40.906414 8037 raft_consensus.cc:2802] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:40.906792 8037 raft_consensus.cc:491] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:40.907176 8037 raft_consensus.cc:3058] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:40.910071 8022 catalog_manager.cc:1261] Loaded cluster ID: 5cb1ee99f0744bcfa6ec1bff323536be
I20250626 19:58:40.915462 7972 tablet_bootstrap.cc:492] T 196180f1e78a4d5ca8aca8d7bb666d42 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:40.924561 7972 tablet_bootstrap.cc:654] T 196180f1e78a4d5ca8aca8d7bb666d42 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:40.926455 8022 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: loading cluster ID for follower catalog manager: success
I20250626 19:58:40.932298 8028 tablet.cc:1722] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250626 19:58:40.944461 7293 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.46.130:40373, user_credentials={real_user=slave}} blocked reactor thread for 65779.1us
I20250626 19:58:40.949148 8035 raft_consensus.cc:491] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:40.951797 8035 raft_consensus.cc:513] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:40.960179 7569 tablet_bootstrap.cc:654] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:40.953655 7972 tablet_bootstrap.cc:492] T 196180f1e78a4d5ca8aca8d7bb666d42 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:40.964432 7972 ts_tablet_manager.cc:1397] T 196180f1e78a4d5ca8aca8d7bb666d42 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.004s
I20250626 19:58:40.969727 7568 tablet_bootstrap.cc:492] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:40.970125 7568 ts_tablet_manager.cc:1397] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.200s user 0.006s sys 0.003s
I20250626 19:58:40.971791 7972 raft_consensus.cc:357] T 196180f1e78a4d5ca8aca8d7bb666d42 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:40.972342 7568 raft_consensus.cc:357] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:40.972936 7568 raft_consensus.cc:383] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:40.973192 7568 raft_consensus.cc:738] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:40.974277 7972 raft_consensus.cc:383] T 196180f1e78a4d5ca8aca8d7bb666d42 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:40.976344 7972 raft_consensus.cc:738] T 196180f1e78a4d5ca8aca8d7bb666d42 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:40.977662 7972 consensus_queue.cc:260] T 196180f1e78a4d5ca8aca8d7bb666d42 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:40.979141 6713 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:40.982116 7214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0487e1d34dd74d70a5b5c8ac8cc0f636" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:40.983058 7214 raft_consensus.cc:2466] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:40.984192 7972 ts_tablet_manager.cc:1428] T 196180f1e78a4d5ca8aca8d7bb666d42 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.019s user 0.000s sys 0.005s
I20250626 19:58:40.992153 7787 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" instance_seqno: 1750967901410941) as {username='slave'} at 127.4.46.129:36355; Asking this server to re-register.
I20250626 19:58:40.984467 7293 leader_election.cc:304] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:40.999161 8036 raft_consensus.cc:2802] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:40.999575 8036 raft_consensus.cc:491] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:40.999948 8036 raft_consensus.cc:3058] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:41.001631 7972 tablet_bootstrap.cc:492] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:40.985132 8035 leader_election.cc:290] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:41.023437 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0487e1d34dd74d70a5b5c8ac8cc0f636" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:41.028791 7119 heartbeater.cc:461] Registering TS with master...
I20250626 19:58:41.035187 8036 raft_consensus.cc:513] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.040027 8037 raft_consensus.cc:513] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:41.041042 7786 ts_manager.cc:194] Registered new tserver with Master: bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129)
I20250626 19:58:41.042353 8037 leader_election.cc:290] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
W20250626 19:58:41.043928 7294 leader_election.cc:343] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:41.043839 7214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d677beae63a49ecbd11ed53e9e421fc" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:41.044364 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0487e1d34dd74d70a5b5c8ac8cc0f636" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:41.044865 7215 raft_consensus.cc:3058] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:41.045866 7069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d677beae63a49ecbd11ed53e9e421fc" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:41.044395 7214 raft_consensus.cc:3058] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:41.051062 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0487e1d34dd74d70a5b5c8ac8cc0f636" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
W20250626 19:58:41.054083 7294 leader_election.cc:343] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
W20250626 19:58:41.054961 7294 leader_election.cc:343] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:41.058189 7214 raft_consensus.cc:2466] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:41.060177 8036 leader_election.cc:290] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:41.050545 7215 raft_consensus.cc:2466] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:41.065732 7972 tablet_bootstrap.cc:654] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:41.066080 7293 leader_election.cc:304] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:41.067327 7569 tablet_bootstrap.cc:492] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:41.067706 7569 ts_tablet_manager.cc:1397] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.201s user 0.010s sys 0.000s
I20250626 19:58:41.067970 8036 raft_consensus.cc:2802] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:41.071208 7569 raft_consensus.cc:357] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:41.071938 7569 raft_consensus.cc:383] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:41.072181 7569 raft_consensus.cc:738] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:41.071801 8036 raft_consensus.cc:695] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:41.072808 7569 consensus_queue.cc:260] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:41.073132 7568 consensus_queue.cc:260] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.075769 7293 leader_election.cc:304] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:41.076762 8037 raft_consensus.cc:2802] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:41.077486 8037 raft_consensus.cc:695] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:41.072983 8036 consensus_queue.cc:237] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.087623 7568 ts_tablet_manager.cc:1428] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.117s user 0.007s sys 0.000s
I20250626 19:58:41.088563 7568 tablet_bootstrap.cc:492] T 2ced063ac30b4168b4af4cf2ba239e25 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:41.079324 8037 consensus_queue.cc:237] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:41.093822 7568 tablet_bootstrap.cc:654] T 2ced063ac30b4168b4af4cf2ba239e25 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:41.102355 6713 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 7b9a0f71535a4a65a775f728edbcc636 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
I20250626 19:58:41.104508 7972 tablet_bootstrap.cc:492] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:41.104938 7972 ts_tablet_manager.cc:1397] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.105s user 0.010s sys 0.000s
I20250626 19:58:41.106583 7569 ts_tablet_manager.cc:1428] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s
I20250626 19:58:41.128865 8022 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: acquiring CA information for follower catalog manager: success
I20250626 19:58:41.131928 7569 tablet_bootstrap.cc:492] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:41.149497 7972 raft_consensus.cc:357] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.150153 7972 raft_consensus.cc:383] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:41.150413 7972 raft_consensus.cc:738] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:41.162573 7569 tablet_bootstrap.cc:654] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:41.162400 7972 consensus_queue.cc:260] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.164824 7972 ts_tablet_manager.cc:1428] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.060s user 0.003s sys 0.003s
I20250626 19:58:41.174990 7568 tablet_bootstrap.cc:492] T 2ced063ac30b4168b4af4cf2ba239e25 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:41.175410 7568 ts_tablet_manager.cc:1397] T 2ced063ac30b4168b4af4cf2ba239e25 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.087s user 0.011s sys 0.001s
I20250626 19:58:41.177661 7568 raft_consensus.cc:357] T 2ced063ac30b4168b4af4cf2ba239e25 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.178717 7568 raft_consensus.cc:383] T 2ced063ac30b4168b4af4cf2ba239e25 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:41.179005 7568 raft_consensus.cc:738] T 2ced063ac30b4168b4af4cf2ba239e25 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:41.179589 7568 consensus_queue.cc:260] T 2ced063ac30b4168b4af4cf2ba239e25 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.191215 7972 tablet_bootstrap.cc:492] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:41.201083 6712 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_daf65ec234534a069c2d1ad34f13952a [id=ee116b84a24e49228c7496e95eaf11bb]
W20250626 19:58:41.204773 6713 catalog_manager.cc:5415] Ignoring report from unknown tablet 089e49e29b2b4f3dbea8a4597d9e36a2
I20250626 19:58:41.211117 7568 ts_tablet_manager.cc:1428] T 2ced063ac30b4168b4af4cf2ba239e25 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.035s user 0.003s sys 0.004s
I20250626 19:58:41.223067 7972 tablet_bootstrap.cc:654] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:41.229133 8047 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:41.229835 8047 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:41.232820 8047 leader_election.cc:290] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417)
W20250626 19:58:41.233731 6717 catalog_manager.cc:5415] Ignoring report from unknown tablet 089e49e29b2b4f3dbea8a4597d9e36a2
I20250626 19:58:41.233582 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 193 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:41.237562 8022 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 19:58:41.237545 6717 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:41.247941 7569 tablet_bootstrap.cc:492] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:41.248380 7569 ts_tablet_manager.cc:1397] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.117s user 0.006s sys 0.003s
I20250626 19:58:41.264065 6713 catalog_manager.cc:5582] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:41.284489 7972 tablet_bootstrap.cc:492] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:41.289232 7972 ts_tablet_manager.cc:1397] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.098s user 0.006s sys 0.006s
I20250626 19:58:41.291896 7972 raft_consensus.cc:357] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.295223 7569 raft_consensus.cc:357] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.295892 7569 raft_consensus.cc:383] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:41.296991 6713 catalog_manager.cc:5582] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } }
I20250626 19:58:41.302866 7569 raft_consensus.cc:738] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:41.307454 7569 consensus_queue.cc:260] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.319178 7972 raft_consensus.cc:383] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:41.322708 7809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 193 } } ignore_live_leader: false dest_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" is_pre_election: true
I20250626 19:58:41.321493 6713 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:41.323573 7809 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6610553ba0df459085f7373935774377 in term 1.
I20250626 19:58:41.325076 6856 leader_election.cc:304] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6610553ba0df459085f7373935774377, 8ae38ec946e14198ae4adc8dde9d66e8; no voters: 188d02ec81e645229acdc5eed98c9f9f
I20250626 19:58:41.325858 8047 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 19:58:41.326007 7569 ts_tablet_manager.cc:1428] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.077s user 0.006s sys 0.001s
I20250626 19:58:41.326185 8047 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 188d02ec81e645229acdc5eed98c9f9f)
I20250626 19:58:41.326493 8047 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:58:41.326932 7569 tablet_bootstrap.cc:492] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:41.337862 8047 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:41.338892 7972 raft_consensus.cc:738] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:41.340389 7972 consensus_queue.cc:260] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.344247 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 193 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f"
I20250626 19:58:41.344266 7809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6610553ba0df459085f7373935774377" candidate_term: 2 candidate_status { last_received { term: 1 index: 193 } } ignore_live_leader: false dest_uuid: "8ae38ec946e14198ae4adc8dde9d66e8"
I20250626 19:58:41.344799 7809 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 1 FOLLOWER]: Advancing to term 2
W20250626 19:58:41.351845 6705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.188:38919, user_credentials={real_user=slave}} blocked reactor thread for 44640.5us
I20250626 19:58:41.359658 6879 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 2 FOLLOWER]: Rejecting Update request from peer 188d02ec81e645229acdc5eed98c9f9f for earlier term 1. Current term is 2. Ops: [1.194-1.194]
I20250626 19:58:41.360122 8047 leader_election.cc:290] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 election: Requested vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417)
I20250626 19:58:41.361030 7809 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6610553ba0df459085f7373935774377 in term 2.
W20250626 19:58:41.361279 6705 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 17d9cd26171943088ad676722edf9577 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 17d9cd26171943088ad676722edf9577 consensus state has no leader
I20250626 19:58:41.364993 6856 leader_election.cc:304] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6610553ba0df459085f7373935774377, 8ae38ec946e14198ae4adc8dde9d66e8; no voters: 188d02ec81e645229acdc5eed98c9f9f
I20250626 19:58:41.366236 8047 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 19:58:41.368168 7569 tablet_bootstrap.cc:654] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:41.372210 7972 ts_tablet_manager.cc:1428] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.083s user 0.005s sys 0.000s
I20250626 19:58:41.373097 8053 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 }, Status: INVALID_TERM, Last received: 1.193, Next index: 194, Last known committed idx: 193, Time since last communication: 0.001s
I20250626 19:58:41.390794 8047 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 2 LEADER]: Becoming Leader. State: Replica: 6610553ba0df459085f7373935774377, State: Running, Role: LEADER
I20250626 19:58:41.391845 8047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 193, Committed index: 193, Last appended: 1.193, Last appended by leader: 193, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:41.402177 7569 tablet_bootstrap.cc:492] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:41.402568 7569 ts_tablet_manager.cc:1397] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.000s
I20250626 19:58:41.404731 7569 raft_consensus.cc:357] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.405339 7569 raft_consensus.cc:383] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:41.405603 7569 raft_consensus.cc:738] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:41.405997 8055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6610553ba0df459085f7373935774377. Latest consensus state: current_term: 2 leader_uuid: "6610553ba0df459085f7373935774377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:41.406136 7569 consensus_queue.cc:260] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.410039 8055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [sys.catalog]: This master's current role is: LEADER
I20250626 19:58:41.416249 8052 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 1 LEADER]: Stepping down as leader of term 1
I20250626 19:58:41.416829 8052 raft_consensus.cc:738] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 188d02ec81e645229acdc5eed98c9f9f, State: Running, Role: LEADER
I20250626 19:58:41.417943 8052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 193, Committed index: 193, Last appended: 1.194, Last appended by leader: 194, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:41.419162 8052 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:58:41.424950 8058 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 19:58:41.432976 7569 ts_tablet_manager.cc:1428] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.030s user 0.004s sys 0.000s
I20250626 19:58:41.433842 7569 tablet_bootstrap.cc:492] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:41.458439 7569 tablet_bootstrap.cc:654] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:41.467238 8058 catalog_manager.cc:671] Loaded metadata for table default.table_acc449d45fc742dab0b039b077cc75e0 [id=11ee0dd5a39742399865b36ea597920d]
I20250626 19:58:41.470805 8058 catalog_manager.cc:671] Loaded metadata for table default.table_63a5ae538eef4f739a22697dfcfb56c7 [id=22351693a59e46a9b9491eb79f1d0fe0]
I20250626 19:58:41.478545 8058 catalog_manager.cc:671] Loaded metadata for table default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]
I20250626 19:58:41.480237 8058 catalog_manager.cc:671] Loaded metadata for table default.table_ced1a26248c748fb8c06cd348d12fb19 [id=3f20c2a68db64ff7afbc98716ecf20b0]
I20250626 19:58:41.482684 7569 tablet_bootstrap.cc:492] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:41.483930 8058 catalog_manager.cc:671] Loaded metadata for table default.table_6bf87f28d4d34102bf1d02e2e1465a33 [id=58d985607f30440ea82dab74e20a1f8f]
I20250626 19:58:41.482935 7980 raft_consensus.cc:491] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:41.484519 7569 ts_tablet_manager.cc:1397] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.000s
I20250626 19:58:41.484536 7980 raft_consensus.cc:513] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:41.487100 7980 leader_election.cc:290] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:41.488904 7359 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9de12a70dc90406c9ca8f58d253f35d5" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:41.489106 7214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9de12a70dc90406c9ca8f58d253f35d5" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:41.490726 7006 leader_election.cc:304] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:41.493376 7980 raft_consensus.cc:2747] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:41.494586 8058 catalog_manager.cc:671] Loaded metadata for table default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a]
I20250626 19:58:41.496450 8058 catalog_manager.cc:671] Loaded metadata for table default.table_fa425acbc3844433a95ac1c8ee1dc26d [id=853420b7ab29422e9683dd9e353f87af]
I20250626 19:58:41.486619 7569 raft_consensus.cc:357] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.497164 7569 raft_consensus.cc:383] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:41.497452 7569 raft_consensus.cc:738] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:41.500779 8036 raft_consensus.cc:491] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:41.500610 7569 consensus_queue.cc:260] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.501220 8036 raft_consensus.cc:513] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.503041 8036 leader_election.cc:290] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:41.505064 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ced063ac30b4168b4af4cf2ba239e25" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:41.505860 7214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ced063ac30b4168b4af4cf2ba239e25" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
W20250626 19:58:41.506253 7294 leader_election.cc:343] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:41.506425 7214 raft_consensus.cc:2466] T 2ced063ac30b4168b4af4cf2ba239e25 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:41.507467 7293 leader_election.cc:304] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:41.508374 8036 raft_consensus.cc:2802] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:41.508689 8036 raft_consensus.cc:491] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:41.508991 8036 raft_consensus.cc:3058] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:41.515939 8036 raft_consensus.cc:513] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.518399 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ced063ac30b4168b4af4cf2ba239e25" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
W20250626 19:58:41.519593 7294 leader_election.cc:343] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Illegal state: must be running to vote when last-logged opid is not known
I20250626 19:58:41.520107 7569 ts_tablet_manager.cc:1428] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20250626 19:58:41.520869 8058 catalog_manager.cc:671] Loaded metadata for table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]
I20250626 19:58:41.524648 8058 catalog_manager.cc:671] Loaded metadata for table default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]
I20250626 19:58:41.540546 7569 tablet_bootstrap.cc:492] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:41.541105 8036 leader_election.cc:290] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:41.541307 7214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ced063ac30b4168b4af4cf2ba239e25" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:41.541823 7214 raft_consensus.cc:3058] T 2ced063ac30b4168b4af4cf2ba239e25 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:40.312240 7814 rpcz_store.cc:276] 0626 19:58:38.903008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:38.903208 (+ 200us) service_pool.cc:225] Handling call
0626 19:58:39.662886 (+759678us) raft_consensus.cc:1469] Updating replica for 158 ops
0626 19:58:39.693691 (+ 30805us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 19:58:39.693745 (+ 54us) raft_consensus.cc:1537] Triggering prepare for 158 ops
0626 19:58:39.694372 (+ 627us) write_op.cc:270] Start()
0626 19:58:39.694503 (+ 131us) write_op.cc:276] Timestamp: P: 1750967906124457 usec, L: 0
0626 19:58:39.818007 (+123504us) write_op.cc:270] Start()
0626 19:58:39.818105 (+ 98us) write_op.cc:276] Timestamp: P: 1750967906159049 usec, L: 0
0626 19:58:39.818863 (+ 758us) write_op.cc:270] Start()
0626 19:58:39.818988 (+ 125us) write_op.cc:276] Timestamp: P: 1750967906254151 usec, L: 0
0626 19:58:39.819516 (+ 528us) write_op.cc:270] Start()
0626 19:58:39.819599 (+ 83us) write_op.cc:276] Timestamp: P: 1750967906265946 usec, L: 0
0626 19:58:39.820086 (+ 487us) write_op.cc:270] Start()
0626 19:58:39.820162 (+ 76us) write_op.cc:276] Timestamp: P: 1750967906280077 usec, L: 0
0626 19:58:39.820597 (+ 435us) write_op.cc:270] Start()
0626 19:58:39.820687 (+ 90us) write_op.cc:276] Timestamp: P: 1750967906379533 usec, L: 0
0626 19:58:39.821131 (+ 444us) write_op.cc:270] Start()
0626 19:58:39.821288 (+ 157us) write_op.cc:276] Timestamp: P: 1750967906436410 usec, L: 0
0626 19:58:39.821751 (+ 463us) write_op.cc:270] Start()
0626 19:58:39.821838 (+ 87us) write_op.cc:276] Timestamp: P: 1750967906477578 usec, L: 0
0626 19:58:39.822304 (+ 466us) write_op.cc:270] Start()
0626 19:58:39.822386 (+ 82us) write_op.cc:276] Timestamp: P: 1750967906535230 usec, L: 0
0626 19:58:39.861728 (+ 39342us) write_op.cc:270] Start()
0626 19:58:39.861842 (+ 114us) write_op.cc:276] Timestamp: P: 1750967906668801 usec, L: 0
0626 19:58:39.862447 (+ 605us) write_op.cc:270] Start()
0626 19:58:39.862548 (+ 101us) write_op.cc:276] Timestamp: P: 1750967906721895 usec, L: 0
0626 19:58:39.863060 (+ 512us) write_op.cc:270] Start()
0626 19:58:39.863174 (+ 114us) write_op.cc:276] Timestamp: P: 1750967906915096 usec, L: 0
0626 19:58:39.863684 (+ 510us) write_op.cc:270] Start()
0626 19:58:39.863768 (+ 84us) write_op.cc:276] Timestamp: P: 1750967906998346 usec, L: 0
0626 19:58:39.877917 (+ 14149us) write_op.cc:270] Start()
0626 19:58:39.878010 (+ 93us) write_op.cc:276] Timestamp: P: 1750967907025263 usec, L: 0
0626 19:58:39.878527 (+ 517us) write_op.cc:270] Start()
0626 19:58:39.890065 (+ 11538us) write_op.cc:276] Timestamp: P: 1750967907126482 usec, L: 0
0626 19:58:39.891056 (+ 991us) write_op.cc:270] Start()
0626 19:58:39.891160 (+ 104us) write_op.cc:276] Timestamp: P: 1750967907174292 usec, L: 0
0626 19:58:39.891692 (+ 532us) write_op.cc:270] Start()
0626 19:58:39.891772 (+ 80us) write_op.cc:276] Timestamp: P: 1750967907203157 usec, L: 0
0626 19:58:39.892230 (+ 458us) write_op.cc:270] Start()
0626 19:58:39.892307 (+ 77us) write_op.cc:276] Timestamp: P: 1750967907299294 usec, L: 0
0626 19:58:39.892751 (+ 444us) write_op.cc:270] Start()
0626 19:58:39.892825 (+ 74us) write_op.cc:276] Timestamp: P: 1750967907623104 usec, L: 0
0626 19:58:39.893259 (+ 434us) write_op.cc:270] Start()
0626 19:58:39.893339 (+ 80us) write_op.cc:276] Timestamp: P: 1750967907661440 usec, L: 0
0626 19:58:39.893770 (+ 431us) write_op.cc:270] Start()
0626 19:58:39.893846 (+ 76us) write_op.cc:276] Timestamp: P: 1750967907789303 usec, L: 0
0626 19:58:39.899534 (+ 5688us) write_op.cc:270] Start()
0626 19:58:39.899614 (+ 80us) write_op.cc:276] Timestamp: P: 1750967907835912 usec, L: 0
0626 19:58:39.900045 (+ 431us) write_op.cc:270] Start()
0626 19:58:39.900112 (+ 67us) write_op.cc:276] Timestamp: P: 1750967907871580 usec, L: 0
0626 19:58:39.900548 (+ 436us) write_op.cc:270] Start()
0626 19:58:39.900657 (+ 109us) write_op.cc:276] Timestamp: P: 1750967907955394 usec, L: 0
0626 19:58:39.901187 (+ 530us) write_op.cc:270] Start()
0626 19:58:39.901278 (+ 91us) write_op.cc:276] Timestamp: P: 1750967908075752 usec, L: 0
0626 19:58:39.901697 (+ 419us) write_op.cc:270] Start()
0626 19:58:39.901774 (+ 77us) write_op.cc:276] Timestamp: P: 1750967908124427 usec, L: 0
0626 19:58:39.902268 (+ 494us) write_op.cc:270] Start()
0626 19:58:39.902342 (+ 74us) write_op.cc:276] Timestamp: P: 1750967908279764 usec, L: 0
0626 19:58:39.902809 (+ 467us) write_op.cc:270] Start()
0626 19:58:39.902888 (+ 79us) write_op.cc:276] Timestamp: P: 1750967908420407 usec, L: 0
0626 19:58:39.903378 (+ 490us) write_op.cc:270] Start()
0626 19:58:39.903455 (+ 77us) write_op.cc:276] Timestamp: P: 1750967908440823 usec, L: 0
0626 19:58:39.912013 (+ 8558us) write_op.cc:270] Start()
0626 19:58:39.912092 (+ 79us) write_op.cc:276] Timestamp: P: 1750967908673671 usec, L: 0
0626 19:58:39.914902 (+ 2810us) write_op.cc:270] Start()
0626 19:58:39.914980 (+ 78us) write_op.cc:276] Timestamp: P: 1750967908969882 usec, L: 0
0626 19:58:39.915423 (+ 443us) write_op.cc:270] Start()
0626 19:58:39.915498 (+ 75us) write_op.cc:276] Timestamp: P: 1750967909030909 usec, L: 0
0626 19:58:39.915981 (+ 483us) write_op.cc:270] Start()
0626 19:58:39.916062 (+ 81us) write_op.cc:276] Timestamp: P: 1750967909057966 usec, L: 0
0626 19:58:39.916512 (+ 450us) write_op.cc:270] Start()
0626 19:58:39.916582 (+ 70us) write_op.cc:276] Timestamp: P: 1750967909108647 usec, L: 0
0626 19:58:39.917016 (+ 434us) write_op.cc:270] Start()
0626 19:58:39.917091 (+ 75us) write_op.cc:276] Timestamp: P: 1750967909181599 usec, L: 0
0626 19:58:39.917521 (+ 430us) write_op.cc:270] Start()
0626 19:58:39.917592 (+ 71us) write_op.cc:276] Timestamp: P: 1750967909221752 usec, L: 0
0626 19:58:39.918043 (+ 451us) write_op.cc:270] Start()
0626 19:58:39.918116 (+ 73us) write_op.cc:276] Timestamp: P: 1750967909244637 usec, L: 0
0626 19:58:39.918548 (+ 432us) write_op.cc:270] Start()
0626 19:58:39.918620 (+ 72us) write_op.cc:276] Timestamp: P: 1750967909498635 usec, L: 0
0626 19:58:39.919136 (+ 516us) write_op.cc:270] Start()
0626 19:58:39.919226 (+ 90us) write_op.cc:276] Timestamp: P: 1750967909531967 usec, L: 0
0626 19:58:39.919679 (+ 453us) write_op.cc:270] Start()
0626 19:58:39.919763 (+ 84us) write_op.cc:276] Timestamp: P: 1750967909595114 usec, L: 0
0626 19:58:39.920348 (+ 585us) write_op.cc:270] Start()
0626 19:58:39.920428 (+ 80us) write_op.cc:276] Timestamp: P: 1750967909630256 usec, L: 0
0626 19:58:39.920863 (+ 435us) write_op.cc:270] Start()
0626 19:58:39.920935 (+ 72us) write_op.cc:276] Timestamp: P: 1750967909780571 usec, L: 0
0626 19:58:39.921339 (+ 404us) write_op.cc:270] Start()
0626 19:58:39.921410 (+ 71us) write_op.cc:276] Timestamp: P: 1750967909859834 usec, L: 0
0626 19:58:39.930016 (+ 8606us) write_op.cc:270] Start()
0626 19:58:39.930151 (+ 135us) write_op.cc:276] Timestamp: P: 1750967910021612 usec, L: 0
0626 19:58:39.930700 (+ 549us) write_op.cc:270] Start()
0626 19:58:39.930803 (+ 103us) write_op.cc:276] Timestamp: P: 1750967910063028 usec, L: 0
0626 19:58:39.931285 (+ 482us) write_op.cc:270] Start()
0626 19:58:39.931355 (+ 70us) write_op.cc:276] Timestamp: P: 1750967910132917 usec, L: 0
0626 19:58:39.931758 (+ 403us) write_op.cc:270] Start()
0626 19:58:39.931828 (+ 70us) write_op.cc:276] Timestamp: P: 1750967910175210 usec, L: 0
0626 19:58:39.932216 (+ 388us) write_op.cc:270] Start()
0626 19:58:39.932362 (+ 146us) write_op.cc:276] Timestamp: P: 1750967910242589 usec, L: 0
0626 19:58:39.932834 (+ 472us) write_op.cc:270] Start()
0626 19:58:39.932923 (+ 89us) write_op.cc:276] Timestamp: P: 1750967910495308 usec, L: 0
0626 19:58:39.933433 (+ 510us) write_op.cc:270] Start()
0626 19:58:39.933517 (+ 84us) write_op.cc:276] Timestamp: P: 1750967910572222 usec, L: 0
0626 19:58:39.934036 (+ 519us) write_op.cc:270] Start()
0626 19:58:39.934111 (+ 75us) write_op.cc:276] Timestamp: P: 1750967910628348 usec, L: 0
0626 19:58:39.934654 (+ 543us) write_op.cc:270] Start()
0626 19:58:39.934764 (+ 110us) write_op.cc:276] Timestamp: P: 1750967910720694 usec, L: 0
0626 19:58:39.935285 (+ 521us) write_op.cc:270] Start()
0626 19:58:39.935357 (+ 72us) write_op.cc:276] Timestamp: P: 1750967910780375 usec, L: 0
0626 19:58:39.935799 (+ 442us) write_op.cc:270] Start()
0626 19:58:39.935884 (+ 85us) write_op.cc:276] Timestamp: P: 1750967910847994 usec, L: 0
0626 19:58:39.947085 (+ 11201us) write_op.cc:270] Start()
0626 19:58:39.947178 (+ 93us) write_op.cc:276] Timestamp: P: 1750967910903228 usec, L: 0
0626 19:58:39.947690 (+ 512us) write_op.cc:270] Start()
0626 19:58:39.947768 (+ 78us) write_op.cc:276] Timestamp: P: 1750967911004747 usec, L: 0
0626 19:58:39.948254 (+ 486us) write_op.cc:270] Start()
0626 19:58:39.948319 (+ 65us) write_op.cc:276] Timestamp: P: 1750967911050348 usec, L: 0
0626 19:58:39.950908 (+ 2589us) write_op.cc:270] Start()
0626 19:58:39.950982 (+ 74us) write_op.cc:276] Timestamp: P: 1750967911128477 usec, L: 0
0626 19:58:39.951439 (+ 457us) write_op.cc:270] Start()
0626 19:58:39.951512 (+ 73us) write_op.cc:276] Timestamp: P: 1750967911169495 usec, L: 0
0626 19:58:39.951998 (+ 486us) write_op.cc:270] Start()
0626 19:58:39.952193 (+ 195us) write_op.cc:276] Timestamp: P: 1750967911336495 usec, L: 0
0626 19:58:39.954053 (+ 1860us) write_op.cc:270] Start()
0626 19:58:39.955737 (+ 1684us) write_op.cc:276] Timestamp: P: 1750967911371851 usec, L: 0
0626 19:58:39.956436 (+ 699us) write_op.cc:270] Start()
0626 19:58:39.956511 (+ 75us) write_op.cc:276] Timestamp: P: 1750967911556271 usec, L: 0
0626 19:58:39.957077 (+ 566us) write_op.cc:270] Start()
0626 19:58:39.957153 (+ 76us) write_op.cc:276] Timestamp: P: 1750967911823355 usec, L: 0
0626 19:58:39.957623 (+ 470us) write_op.cc:270] Start()
0626 19:58:39.957695 (+ 72us) write_op.cc:276] Timestamp: P: 1750967911843852 usec, L: 0
0626 19:58:39.958354 (+ 659us) write_op.cc:270] Start()
0626 19:58:39.958447 (+ 93us) write_op.cc:276] Timestamp: P: 1750967911950460 usec, L: 0
0626 19:58:39.958967 (+ 520us) write_op.cc:270] Start()
0626 19:58:39.959039 (+ 72us) write_op.cc:276] Timestamp: P: 1750967912019122 usec, L: 0
0626 19:58:39.959502 (+ 463us) write_op.cc:270] Start()
0626 19:58:39.959579 (+ 77us) write_op.cc:276] Timestamp: P: 1750967912064790 usec, L: 0
0626 19:58:39.959986 (+ 407us) write_op.cc:270] Start()
0626 19:58:39.960055 (+ 69us) write_op.cc:276] Timestamp: P: 1750967912087171 usec, L: 0
0626 19:58:39.960470 (+ 415us) write_op.cc:270] Start()
0626 19:58:39.960538 (+ 68us) write_op.cc:276] Timestamp: P: 1750967912237579 usec, L: 0
0626 19:58:39.960906 (+ 368us) write_op.cc:270] Start()
0626 19:58:39.960977 (+ 71us) write_op.cc:276] Timestamp: P: 1750967912415214 usec, L: 0
0626 19:58:39.961391 (+ 414us) write_op.cc:270] Start()
0626 19:58:39.961468 (+ 77us) write_op.cc:276] Timestamp: P: 1750967912447257 usec, L: 0
0626 19:58:39.961978 (+ 510us) write_op.cc:270] Start()
0626 19:58:39.962051 (+ 73us) write_op.cc:276] Timestamp: P: 1750967912469533 usec, L: 0
0626 19:58:39.962476 (+ 425us) write_op.cc:270] Start()
0626 19:58:39.962546 (+ 70us) write_op.cc:276] Timestamp: P: 1750967912472633 usec, L: 0
0626 19:58:39.965806 (+ 3260us) write_op.cc:270] Start()
0626 19:58:39.965891 (+ 85us) write_op.cc:276] Timestamp: P: 1750967912595902 usec, L: 0
0626 19:58:39.966369 (+ 478us) write_op.cc:270] Start()
0626 19:58:39.966445 (+ 76us) write_op.cc:276] Timestamp: P: 1750967912693190 usec, L: 0
0626 19:58:39.966901 (+ 456us) write_op.cc:270] Start()
0626 19:58:39.966982 (+ 81us) write_op.cc:276] Timestamp: P: 1750967912753588 usec, L: 0
0626 19:58:39.967446 (+ 464us) write_op.cc:270] Start()
0626 19:58:39.967516 (+ 70us) write_op.cc:276] Timestamp: P: 1750967912800779 usec, L: 0
0626 19:58:39.967942 (+ 426us) write_op.cc:270] Start()
0626 19:58:39.968016 (+ 74us) write_op.cc:276] Timestamp: P: 1750967912809255 usec, L: 0
0626 19:58:39.968485 (+ 469us) write_op.cc:270] Start()
0626 19:58:39.968554 (+ 69us) write_op.cc:276] Timestamp: P: 1750967912931682 usec, L: 0
0626 19:58:39.969008 (+ 454us) write_op.cc:270] Start()
0626 19:58:39.969083 (+ 75us) write_op.cc:276] Timestamp: P: 1750967913048478 usec, L: 0
0626 19:58:39.969535 (+ 452us) write_op.cc:270] Start()
0626 19:58:39.969608 (+ 73us) write_op.cc:276] Timestamp: P: 1750967913060129 usec, L: 0
0626 19:58:39.970003 (+ 395us) write_op.cc:270] Start()
0626 19:58:39.970075 (+ 72us) write_op.cc:276] Timestamp: P: 1750967913124092 usec, L: 0
0626 19:58:39.970496 (+ 421us) write_op.cc:270] Start()
0626 19:58:39.970564 (+ 68us) write_op.cc:276] Timestamp: P: 1750967913154174 usec, L: 0
0626 19:58:39.975077 (+ 4513us) write_op.cc:270] Start()
0626 19:58:39.975159 (+ 82us) write_op.cc:276] Timestamp: P: 1750967913202979 usec, L: 0
0626 19:58:39.975617 (+ 458us) write_op.cc:270] Start()
0626 19:58:39.975696 (+ 79us) write_op.cc:276] Timestamp: P: 1750967913260156 usec, L: 0
0626 19:58:39.976184 (+ 488us) write_op.cc:270] Start()
0626 19:58:39.976265 (+ 81us) write_op.cc:276] Timestamp: P: 1750967913374165 usec, L: 0
0626 19:58:39.976716 (+ 451us) write_op.cc:270] Start()
0626 19:58:39.976791 (+ 75us) write_op.cc:276] Timestamp: P: 1750967913460590 usec, L: 0
0626 19:58:39.977216 (+ 425us) write_op.cc:270] Start()
0626 19:58:39.977296 (+ 80us) write_op.cc:276] Timestamp: P: 1750967913511372 usec, L: 0
0626 19:58:39.977725 (+ 429us) write_op.cc:270] Start()
0626 19:58:39.977801 (+ 76us) write_op.cc:276] Timestamp: P: 1750967913660575 usec, L: 0
0626 19:58:40.000153 (+ 22352us) write_op.cc:270] Start()
0626 19:58:40.002683 (+ 2530us) write_op.cc:276] Timestamp: P: 1750967913758964 usec, L: 0
0626 19:58:40.003341 (+ 658us) write_op.cc:270] Start()
0626 19:58:40.003435 (+ 94us) write_op.cc:276] Timestamp: P: 1750967913825169 usec, L: 0
0626 19:58:40.015052 (+ 11617us) write_op.cc:270] Start()
0626 19:58:40.015159 (+ 107us) write_op.cc:276] Timestamp: P: 1750967913915304 usec, L: 0
0626 19:58:40.015731 (+ 572us) write_op.cc:270] Start()
0626 19:58:40.015816 (+ 85us) write_op.cc:276] Timestamp: P: 1750967913983725 usec, L: 0
0626 19:58:40.016289 (+ 473us) write_op.cc:270] Start()
0626 19:58:40.016363 (+ 74us) write_op.cc:276] Timestamp: P: 1750967914087058 usec, L: 0
0626 19:58:40.016811 (+ 448us) write_op.cc:270] Start()
0626 19:58:40.016884 (+ 73us) write_op.cc:276] Timestamp: P: 1750967914148105 usec, L: 0
0626 19:58:40.017423 (+ 539us) write_op.cc:270] Start()
0626 19:58:40.017498 (+ 75us) write_op.cc:276] Timestamp: P: 1750967914204726 usec, L: 0
0626 19:58:40.017955 (+ 457us) write_op.cc:270] Start()
0626 19:58:40.018024 (+ 69us) write_op.cc:276] Timestamp: P: 1750967914326447 usec, L: 0
0626 19:58:40.018421 (+ 397us) write_op.cc:270] Start()
0626 19:58:40.018602 (+ 181us) write_op.cc:276] Timestamp: P: 1750967914357758 usec, L: 0
0626 19:58:40.019153 (+ 551us) write_op.cc:270] Start()
0626 19:58:40.019244 (+ 91us) write_op.cc:276] Timestamp: P: 1750967914428873 usec, L: 0
0626 19:58:40.021424 (+ 2180us) write_op.cc:270] Start()
0626 19:58:40.021499 (+ 75us) write_op.cc:276] Timestamp: P: 1750967914479677 usec, L: 0
0626 19:58:40.021936 (+ 437us) write_op.cc:270] Start()
0626 19:58:40.022011 (+ 75us) write_op.cc:276] Timestamp: P: 1750967914520660 usec, L: 0
0626 19:58:40.022456 (+ 445us) write_op.cc:270] Start()
0626 19:58:40.022529 (+ 73us) write_op.cc:276] Timestamp: P: 1750967914590213 usec, L: 0
0626 19:58:40.022994 (+ 465us) write_op.cc:270] Start()
0626 19:58:40.023067 (+ 73us) write_op.cc:276] Timestamp: P: 1750967914630913 usec, L: 0
0626 19:58:40.023628 (+ 561us) write_op.cc:270] Start()
0626 19:58:40.023709 (+ 81us) write_op.cc:276] Timestamp: P: 1750967914648720 usec, L: 0
0626 19:58:40.024223 (+ 514us) write_op.cc:270] Start()
0626 19:58:40.024296 (+ 73us) write_op.cc:276] Timestamp: P: 1750967914703486 usec, L: 0
0626 19:58:40.024898 (+ 602us) write_op.cc:270] Start()
0626 19:58:40.024973 (+ 75us) write_op.cc:276] Timestamp: P: 1750967914734470 usec, L: 0
0626 19:58:40.025441 (+ 468us) write_op.cc:270] Start()
0626 19:58:40.025533 (+ 92us) write_op.cc:276] Timestamp: P: 1750967914932922 usec, L: 0
0626 19:58:40.025947 (+ 414us) write_op.cc:270] Start()
0626 19:58:40.026015 (+ 68us) write_op.cc:276] Timestamp: P: 1750967915117736 usec, L: 0
0626 19:58:40.026464 (+ 449us) write_op.cc:270] Start()
0626 19:58:40.026537 (+ 73us) write_op.cc:276] Timestamp: P: 1750967915130920 usec, L: 0
0626 19:58:40.027002 (+ 465us) write_op.cc:270] Start()
0626 19:58:40.027079 (+ 77us) write_op.cc:276] Timestamp: P: 1750967915188313 usec, L: 0
0626 19:58:40.027575 (+ 496us) write_op.cc:270] Start()
0626 19:58:40.027653 (+ 78us) write_op.cc:276] Timestamp: P: 1750967915201314 usec, L: 0
0626 19:58:40.028093 (+ 440us) write_op.cc:270] Start()
0626 19:58:40.028179 (+ 86us) write_op.cc:276] Timestamp: P: 1750967915241164 usec, L: 0
0626 19:58:40.028627 (+ 448us) write_op.cc:270] Start()
0626 19:58:40.028698 (+ 71us) write_op.cc:276] Timestamp: P: 1750967915272665 usec, L: 0
0626 19:58:40.029073 (+ 375us) write_op.cc:270] Start()
0626 19:58:40.029139 (+ 66us) write_op.cc:276] Timestamp: P: 1750967915398192 usec, L: 0
0626 19:58:40.029558 (+ 419us) write_op.cc:270] Start()
0626 19:58:40.029634 (+ 76us) write_op.cc:276] Timestamp: P: 1750967915544156 usec, L: 0
0626 19:58:40.030063 (+ 429us) write_op.cc:270] Start()
0626 19:58:40.030132 (+ 69us) write_op.cc:276] Timestamp: P: 1750967915584176 usec, L: 0
0626 19:58:40.030547 (+ 415us) write_op.cc:270] Start()
0626 19:58:40.030619 (+ 72us) write_op.cc:276] Timestamp: P: 1750967915624784 usec, L: 0
0626 19:58:40.031079 (+ 460us) write_op.cc:270] Start()
0626 19:58:40.031150 (+ 71us) write_op.cc:276] Timestamp: P: 1750967915842859 usec, L: 0
0626 19:58:40.031592 (+ 442us) write_op.cc:270] Start()
0626 19:58:40.031663 (+ 71us) write_op.cc:276] Timestamp: P: 1750967915872496 usec, L: 0
0626 19:58:40.032054 (+ 391us) write_op.cc:270] Start()
0626 19:58:40.032124 (+ 70us) write_op.cc:276] Timestamp: P: 1750967915947467 usec, L: 0
0626 19:58:40.032530 (+ 406us) write_op.cc:270] Start()
0626 19:58:40.032600 (+ 70us) write_op.cc:276] Timestamp: P: 1750967916013727 usec, L: 0
0626 19:58:40.032984 (+ 384us) write_op.cc:270] Start()
0626 19:58:40.033053 (+ 69us) write_op.cc:276] Timestamp: P: 1750967916110361 usec, L: 0
0626 19:58:40.033474 (+ 421us) write_op.cc:270] Start()
0626 19:58:40.033554 (+ 80us) write_op.cc:276] Timestamp: P: 1750967916179404 usec, L: 0
0626 19:58:40.033939 (+ 385us) write_op.cc:270] Start()
0626 19:58:40.034006 (+ 67us) write_op.cc:276] Timestamp: P: 1750967916200472 usec, L: 0
0626 19:58:40.034464 (+ 458us) write_op.cc:270] Start()
0626 19:58:40.034536 (+ 72us) write_op.cc:276] Timestamp: P: 1750967916266389 usec, L: 0
0626 19:58:40.035511 (+ 975us) write_op.cc:270] Start()
0626 19:58:40.035587 (+ 76us) write_op.cc:276] Timestamp: P: 1750967916349812 usec, L: 0
0626 19:58:40.036030 (+ 443us) write_op.cc:270] Start()
0626 19:58:40.036111 (+ 81us) write_op.cc:276] Timestamp: P: 1750967916372028 usec, L: 0
0626 19:58:40.036553 (+ 442us) write_op.cc:270] Start()
0626 19:58:40.036621 (+ 68us) write_op.cc:276] Timestamp: P: 1750967916454952 usec, L: 0
0626 19:58:40.037301 (+ 680us) write_op.cc:270] Start()
0626 19:58:40.037377 (+ 76us) write_op.cc:276] Timestamp: P: 1750967916501930 usec, L: 0
0626 19:58:40.037921 (+ 544us) write_op.cc:270] Start()
0626 19:58:40.038000 (+ 79us) write_op.cc:276] Timestamp: P: 1750967916572191 usec, L: 0
0626 19:58:40.038620 (+ 620us) write_op.cc:270] Start()
0626 19:58:40.039173 (+ 553us) write_op.cc:276] Timestamp: P: 1750967916641436 usec, L: 0
0626 19:58:40.039669 (+ 496us) write_op.cc:270] Start()
0626 19:58:40.039754 (+ 85us) write_op.cc:276] Timestamp: P: 1750967916698731 usec, L: 0
0626 19:58:40.040191 (+ 437us) write_op.cc:270] Start()
0626 19:58:40.040267 (+ 76us) write_op.cc:276] Timestamp: P: 1750967916783398 usec, L: 0
0626 19:58:40.040697 (+ 430us) write_op.cc:270] Start()
0626 19:58:40.040765 (+ 68us) write_op.cc:276] Timestamp: P: 1750967916823405 usec, L: 0
0626 19:58:40.041237 (+ 472us) write_op.cc:270] Start()
0626 19:58:40.041327 (+ 90us) write_op.cc:276] Timestamp: P: 1750967916930115 usec, L: 0
0626 19:58:40.041747 (+ 420us) write_op.cc:270] Start()
0626 19:58:40.041819 (+ 72us) write_op.cc:276] Timestamp: P: 1750967916995141 usec, L: 0
0626 19:58:40.042254 (+ 435us) write_op.cc:270] Start()
0626 19:58:40.042328 (+ 74us) write_op.cc:276] Timestamp: P: 1750967917179757 usec, L: 0
0626 19:58:40.042780 (+ 452us) write_op.cc:270] Start()
0626 19:58:40.042855 (+ 75us) write_op.cc:276] Timestamp: P: 1750967917214564 usec, L: 0
0626 19:58:40.043965 (+ 1110us) write_op.cc:270] Start()
0626 19:58:40.044050 (+ 85us) write_op.cc:276] Timestamp: P: 1750967917314545 usec, L: 0
0626 19:58:40.044503 (+ 453us) write_op.cc:270] Start()
0626 19:58:40.044597 (+ 94us) write_op.cc:276] Timestamp: P: 1750967917415346 usec, L: 0
0626 19:58:40.045023 (+ 426us) write_op.cc:270] Start()
0626 19:58:40.045097 (+ 74us) write_op.cc:276] Timestamp: P: 1750967917446381 usec, L: 0
0626 19:58:40.045552 (+ 455us) write_op.cc:270] Start()
0626 19:58:40.045624 (+ 72us) write_op.cc:276] Timestamp: P: 1750967917491985 usec, L: 0
0626 19:58:40.046384 (+ 760us) write_op.cc:270] Start()
0626 19:58:40.046459 (+ 75us) write_op.cc:276] Timestamp: P: 1750967917525062 usec, L: 0
0626 19:58:40.046991 (+ 532us) write_op.cc:270] Start()
0626 19:58:40.047069 (+ 78us) write_op.cc:276] Timestamp: P: 1750967917609478 usec, L: 0
0626 19:58:40.048529 (+ 1460us) write_op.cc:270] Start()
0626 19:58:40.048601 (+ 72us) write_op.cc:276] Timestamp: P: 1750967917659971 usec, L: 0
0626 19:58:40.049014 (+ 413us) write_op.cc:270] Start()
0626 19:58:40.049083 (+ 69us) write_op.cc:276] Timestamp: P: 1750967917718798 usec, L: 0
0626 19:58:40.049534 (+ 451us) write_op.cc:270] Start()
0626 19:58:40.049601 (+ 67us) write_op.cc:276] Timestamp: P: 1750967917809630 usec, L: 0
0626 19:58:40.050056 (+ 455us) write_op.cc:270] Start()
0626 19:58:40.050165 (+ 109us) write_op.cc:276] Timestamp: P: 1750967917844000 usec, L: 0
0626 19:58:40.050620 (+ 455us) write_op.cc:270] Start()
0626 19:58:40.050690 (+ 70us) write_op.cc:276] Timestamp: P: 1750967917897502 usec, L: 0
0626 19:58:40.051197 (+ 507us) write_op.cc:270] Start()
0626 19:58:40.051269 (+ 72us) write_op.cc:276] Timestamp: P: 1750967917970903 usec, L: 0
0626 19:58:40.051700 (+ 431us) write_op.cc:270] Start()
0626 19:58:40.051774 (+ 74us) write_op.cc:276] Timestamp: P: 1750967918063646 usec, L: 0
0626 19:58:40.052199 (+ 425us) write_op.cc:270] Start()
0626 19:58:40.052282 (+ 83us) write_op.cc:276] Timestamp: P: 1750967918300884 usec, L: 0
0626 19:58:40.052899 (+ 617us) write_op.cc:270] Start()
0626 19:58:40.052979 (+ 80us) write_op.cc:276] Timestamp: P: 1750967918322858 usec, L: 0
0626 19:58:40.053427 (+ 448us) write_op.cc:270] Start()
0626 19:58:40.053499 (+ 72us) write_op.cc:276] Timestamp: P: 1750967918475294 usec, L: 0
0626 19:58:40.053917 (+ 418us) write_op.cc:270] Start()
0626 19:58:40.053988 (+ 71us) write_op.cc:276] Timestamp: P: 1750967918505393 usec, L: 0
0626 19:58:40.054398 (+ 410us) write_op.cc:270] Start()
0626 19:58:40.054494 (+ 96us) write_op.cc:276] Timestamp: P: 1750967918513330 usec, L: 0
0626 19:58:40.054903 (+ 409us) write_op.cc:270] Start()
0626 19:58:40.054974 (+ 71us) write_op.cc:276] Timestamp: P: 1750967918515422 usec, L: 0
0626 19:58:40.055407 (+ 433us) write_op.cc:270] Start()
0626 19:58:40.055480 (+ 73us) write_op.cc:276] Timestamp: P: 1750967918527718 usec, L: 0
0626 19:58:40.074115 (+ 18635us) log.cc:844] Serialized 87044 byte log entry
0626 19:58:40.080258 (+ 6143us) raft_consensus.cc:1643] Marking committed up to 161
0626 19:58:40.118071 (+ 37813us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:40.118091 (+ 20us) write_op.cc:454] Released schema lock
0626 19:58:40.149566 (+ 31475us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:40.149586 (+ 20us) write_op.cc:454] Released schema lock
0626 19:58:40.220678 (+ 71092us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 19:58:40.226345 (+ 5667us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b54000703d4. stack: 00007f65c2a6d483 00007f65c2a6d380 00007f65c25e9574 00000000004fa174 00000000004fa109 00007f65c996fa70 00007f65c99f3b1a 00007f65cb7d111c 00007f65cb7d972c 00007f65cb7d96d8 00007f65cb7d9658 00007f65cb7d9613 00007f65cb7d88bc 00007f65c9e2b8f6 00007f65c9e2b888 00007f65c9a0a17a
0626 19:58:40.267554 (+ 41209us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 19:58:40.310862 (+ 43308us) raft_consensus.cc:1683] finished
0626 19:58:40.310979 (+ 117us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 19:58:40.311499 (+ 520us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:39.815383 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:39.815877 (+ 494us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:39.815912 (+ 35us) write_op.cc:435] Acquired schema lock
0626 19:58:39.815930 (+ 18us) tablet.cc:660] Decoding operations
0626 19:58:39.816277 (+ 347us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:39.816349 (+ 72us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:39.816374 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:39.816590 (+ 216us) tablet.cc:699] Row locks acquired
0626 19:58:39.816606 (+ 16us) write_op.cc:260] PREPARE: finished
0626 19:58:40.080458 (+263852us) op_driver.cc:464] REPLICATION: finished
0626 19:58:40.115827 (+ 35369us) write_op.cc:301] APPLY: starting
0626 19:58:40.115890 (+ 63us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:40.116068 (+ 178us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:40.116094 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:40.116483 (+ 389us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:40.116725 (+ 242us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:40.116860 (+ 135us) write_op.cc:312] APPLY: finished
0626 19:58:40.117155 (+ 295us) log.cc:844] Serialized 28 byte log entry
0626 19:58:40.117556 (+ 401us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:40.117627 (+ 71us) write_op.cc:454] Released schema lock
0626 19:58:40.117675 (+ 48us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0626 19:58:39.834973 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:39.835432 (+ 459us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:39.835458 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:58:39.835473 (+ 15us) tablet.cc:660] Decoding operations
0626 19:58:39.835765 (+ 292us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:39.835847 (+ 82us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:39.835872 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:39.836079 (+ 207us) tablet.cc:699] Row locks acquired
0626 19:58:39.836090 (+ 11us) write_op.cc:260] PREPARE: finished
0626 19:58:40.118334 (+282244us) op_driver.cc:464] REPLICATION: finished
0626 19:58:40.145211 (+ 26877us) write_op.cc:301] APPLY: starting
0626 19:58:40.145270 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:40.145364 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:40.145380 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:40.145770 (+ 390us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:40.145896 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 19:58:40.146023 (+ 127us) write_op.cc:312] APPLY: finished
0626 19:58:40.146218 (+ 195us) log.cc:844] Serialized 28 byte log entry
0626 19:58:40.146557 (+ 339us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:40.150827 (+ 4270us) write_op.cc:454] Released schema lock
0626 19:58:40.150858 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0626 19:58:40.151043 (+ 185us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:40.151069 (+ 26us) write_op.cc:454] Released schema lock
Related trace 'op':
0626 19:58:39.836340 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:39.836736 (+ 396us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:39.836761 (+ 25us) write_op.cc:435] Acquired schema lock
0626 19:58:39.836776 (+ 15us) tablet.cc:660] Decoding operations
0626 19:58:39.837037 (+ 261us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:39.837087 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:39.837111 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:39.837317 (+ 206us) tablet.cc:699] Row locks acquired
0626 19:58:39.837332 (+ 15us) write_op.cc:260] PREPARE: finished
0626 19:58:40.118548 (+281216us) op_driver.cc:464] REPLICATION: finished
0626 19:58:40.143824 (+ 25276us) write_op.cc:301] APPLY: starting
0626 19:58:40.143881 (+ 57us) tablet.
I20250626 19:58:41.552064 7214 raft_consensus.cc:2466] T 2ced063ac30b4168b4af4cf2ba239e25 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:41.554109 7293 leader_election.cc:304] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:41.556349 8058 catalog_manager.cc:671] Loaded metadata for table default.table_6f04f2a6bf414cb8912fe28aff625d2e [id=912c0fafb9264e07b1fa485ea0d193d9]
I20250626 19:58:41.559752 8058 catalog_manager.cc:671] Loaded metadata for table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]
I20250626 19:58:41.561472 8058 catalog_manager.cc:671] Loaded metadata for table default.table_dc0e9650e86e4c60aec4f82f9b82f959 [id=d1d807f596fd4519b4d74850a971205a]
I20250626 19:58:41.563007 7952 raft_consensus.cc:2802] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:41.563477 7952 raft_consensus.cc:695] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:41.564239 7952 consensus_queue.cc:237] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.569507 8058 catalog_manager.cc:671] Loaded metadata for table default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]
I20250626 19:58:41.571537 8058 catalog_manager.cc:671] Loaded metadata for table default.table_daf65ec234534a069c2d1ad34f13952a [id=ee116b84a24e49228c7496e95eaf11bb]
I20250626 19:58:41.581939 8058 catalog_manager.cc:671] Loaded metadata for table default.table_a43e8ff4d1844c0b9dddf28fc310fd8f [id=ef672ba275b54933ae739bf9e560ae68]
I20250626 19:58:41.585228 8011 raft_consensus.cc:491] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:41.585707 8011 raft_consensus.cc:513] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.587554 8011 leader_election.cc:290] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:41.589183 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5646bc4c82d4c23941b27994648ee2e" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:41.589762 7073 raft_consensus.cc:2466] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:41.590967 7150 leader_election.cc:304] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:41.592695 8011 raft_consensus.cc:2802] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:41.592689 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5646bc4c82d4c23941b27994648ee2e" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:41.593094 8011 raft_consensus.cc:491] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:41.593230 7360 raft_consensus.cc:2466] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:41.593423 8011 raft_consensus.cc:3058] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:41.603441 8058 tablet_loader.cc:96] loaded metadata for tablet 0487e1d34dd74d70a5b5c8ac8cc0f636 (table default.table_6f04f2a6bf414cb8912fe28aff625d2e [id=912c0fafb9264e07b1fa485ea0d193d9])
I20250626 19:58:41.605058 8058 tablet_loader.cc:96] loaded metadata for tablet 196180f1e78a4d5ca8aca8d7bb666d42 (table default.table_dc0e9650e86e4c60aec4f82f9b82f959 [id=d1d807f596fd4519b4d74850a971205a])
I20250626 19:58:41.606423 8058 tablet_loader.cc:96] loaded metadata for tablet 2ced063ac30b4168b4af4cf2ba239e25 (table default.table_acc449d45fc742dab0b039b077cc75e0 [id=11ee0dd5a39742399865b36ea597920d])
I20250626 19:58:41.607811 8058 tablet_loader.cc:96] loaded metadata for tablet 40b6570e38434e0c8301d010e96c67a3 (table default.table_ced1a26248c748fb8c06cd348d12fb19 [id=3f20c2a68db64ff7afbc98716ecf20b0])
I20250626 19:58:41.609300 8058 tablet_loader.cc:96] loaded metadata for tablet 55fad5c8639047a0a5519243d6ae76a3 (table default.table_6bf87f28d4d34102bf1d02e2e1465a33 [id=58d985607f30440ea82dab74e20a1f8f])
I20250626 19:58:41.610603 8058 tablet_loader.cc:96] loaded metadata for tablet 60a00cf0b1854a20b54c4ae8a59ede29 (table default.table_daf65ec234534a069c2d1ad34f13952a [id=ee116b84a24e49228c7496e95eaf11bb])
I20250626 19:58:41.612767 8058 tablet_loader.cc:96] loaded metadata for tablet 7b9a0f71535a4a65a775f728edbcc636 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49])
I20250626 19:58:41.614029 8058 tablet_loader.cc:96] loaded metadata for tablet 7bf0c724d4684773b40aff24524e1e92 (table default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d])
I20250626 19:58:41.615326 8058 tablet_loader.cc:96] loaded metadata for tablet 83493d647cc8464e8e650bbca7c6072c (table default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a])
I20250626 19:58:41.616710 8058 tablet_loader.cc:96] loaded metadata for tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7])
I20250626 19:58:41.617961 8058 tablet_loader.cc:96] loaded metadata for tablet 9de12a70dc90406c9ca8f58d253f35d5 (table default.table_fa425acbc3844433a95ac1c8ee1dc26d [id=853420b7ab29422e9683dd9e353f87af])
I20250626 19:58:41.619510 8058 tablet_loader.cc:96] loaded metadata for tablet a5646bc4c82d4c23941b27994648ee2e (table default.table_63a5ae538eef4f739a22697dfcfb56c7 [id=22351693a59e46a9b9491eb79f1d0fe0])
I20250626 19:58:41.620777 8058 tablet_loader.cc:96] loaded metadata for tablet c7159554b45442b5aa5069d7a47d1f8e (table default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a])
I20250626 19:58:41.622961 8058 tablet_loader.cc:96] loaded metadata for tablet f9adef69fe7442a4836a84934e792c05 (table default.table_a43e8ff4d1844c0b9dddf28fc310fd8f [id=ef672ba275b54933ae739bf9e560ae68])
I20250626 19:58:41.624465 8058 tablet_loader.cc:96] loaded metadata for tablet fabf0275e32e400c9731eb09d9c2620c (table default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21])
I20250626 19:58:41.627272 8058 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 19:58:41.627341 8011 raft_consensus.cc:513] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.630972 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5646bc4c82d4c23941b27994648ee2e" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:41.631510 7360 raft_consensus.cc:3058] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:41.637883 7360 raft_consensus.cc:2466] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:41.638978 7152 leader_election.cc:304] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:41.639744 7932 raft_consensus.cc:2802] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:41.640193 7932 raft_consensus.cc:695] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:41.640975 7932 consensus_queue.cc:237] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.645076 8058 catalog_manager.cc:1261] Loaded cluster ID: 5cb1ee99f0744bcfa6ec1bff323536be
I20250626 19:58:41.645471 8058 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 19:58:41.672453 8058 catalog_manager.cc:1506] Loading token signing keys...
I20250626 19:58:41.625540 7073 raft_consensus.cc:3058] T 9de12a70dc90406c9ca8f58d253f35d5 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:41.629954 7069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5646bc4c82d4c23941b27994648ee2e" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:41.683389 7069 raft_consensus.cc:3058] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:41.687036 8058 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: Loaded TSK: 0
I20250626 19:58:41.688333 8058 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 19:58:41.690435 7069 raft_consensus.cc:2466] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:41.606714 7569 tablet_bootstrap.cc:654] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:41.698367 8011 leader_election.cc:290] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:41.702702 8058 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250626 19:58:41.715641 8058 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 28
I20250626 19:58:41.723325 7980 raft_consensus.cc:491] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:41.723798 7980 raft_consensus.cc:513] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.725682 7980 leader_election.cc:290] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:41.727362 7214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "196180f1e78a4d5ca8aca8d7bb666d42" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:41.727967 7214 raft_consensus.cc:2466] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:41.729578 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "196180f1e78a4d5ca8aca8d7bb666d42" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:41.730094 7360 raft_consensus.cc:2466] T 196180f1e78a4d5ca8aca8d7bb666d42 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:41.731274 7006 leader_election.cc:304] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:41.739146 7932 raft_consensus.cc:491] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:41.739595 7932 raft_consensus.cc:513] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:41.741400 7932 leader_election.cc:290] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:41.744967 7359 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e267bd75d7d342a3a6c28b17eb37d83e" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:41.745527 7359 raft_consensus.cc:2466] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:41.747277 7152 leader_election.cc:304] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
W20250626 19:58:41.751307 7150 leader_election.cc:343] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e
I20250626 19:58:41.754343 7983 raft_consensus.cc:2802] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 19:58:41.754683 7983 raft_consensus.cc:491] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:41.755007 7983 raft_consensus.cc:3058] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:58:41.761337 7980 raft_consensus.cc:2802] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:41.761950 7980 raft_consensus.cc:491] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:41.762344 7980 raft_consensus.cc:3058] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:41.770493 7118 heartbeater.cc:499] Master 127.4.46.188:38919 was elected leader, sending a full tablet report...
I20250626 19:58:41.779727 7983 raft_consensus.cc:513] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 19:58:41.791226 7150 leader_election.cc:343] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e
I20250626 19:58:41.792685 6718 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]
I20250626 19:58:41.806602 7814 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 2 FOLLOWER]: Refusing update from remote peer 6610553ba0df459085f7373935774377: Log matching property violated. Preceding OpId in replica: term: 1 index: 162. Preceding OpId from leader: term: 2 index: 194. (index mismatch)
I20250626 19:58:41.807608 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e267bd75d7d342a3a6c28b17eb37d83e" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:41.808120 7360 raft_consensus.cc:3058] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 19:58:41.811545 8055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 194, Last known committed idx: 161, Time since last communication: 0.000s
I20250626 19:58:41.841897 7360 raft_consensus.cc:2466] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 2.
I20250626 19:58:41.859362 7980 raft_consensus.cc:513] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:41.871753 7569 tablet_bootstrap.cc:492] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:41.872418 7569 ts_tablet_manager.cc:1397] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.332s user 0.006s sys 0.005s
I20250626 19:58:41.876402 7152 leader_election.cc:304] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters: bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:41.877753 8011 raft_consensus.cc:2802] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 2 FOLLOWER]: Leader election won for term 2
I20250626 19:58:41.879935 8011 raft_consensus.cc:695] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [term 2 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:41.880754 8011 consensus_queue.cc:237] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:41.883688 7983 leader_election.cc:290] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 2 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:41.896478 8066 raft_consensus.cc:491] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:41.897055 8066 raft_consensus.cc:513] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:41.900708 7214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d677beae63a49ecbd11ed53e9e421fc" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:41.901458 7214 raft_consensus.cc:2391] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbe151be01e24c82854d9c8c79c797ca in current term 1: Already voted for candidate 297d2c913667408bae82668a557e06c7 in this term.
I20250626 19:58:41.911229 6728 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 2 FOLLOWER]: Refusing update from remote peer 6610553ba0df459085f7373935774377: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 2 index: 194. (term mismatch)
I20250626 19:58:41.911589 6728 pending_rounds.cc:85] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Aborting all ops after (but not including) 193
I20250626 19:58:41.911852 6728 pending_rounds.cc:107] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: Aborting uncommitted WRITE_OP operation due to leader change: 1.194
E20250626 19:58:41.912811 6713 catalog_manager.cc:5737] Error updating tablets from 297d2c913667408bae82668a557e06c7: Aborted: Op aborted by new leader. Tablet report was: is_incremental: true updated_tablets { tablet_id: "2ced063ac30b4168b4af4cf2ba239e25" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "55fad5c8639047a0a5519243d6ae76a3" state: RUNNING consensus_state { current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27353 live_row_count: 0 } } updated_tablets { tablet_id: "9d677beae63a49ecbd11ed53e9e421fc" state: RUNNING consensus_state { current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "0487e1d34dd74d70a5b5c8ac8cc0f636" state: RUNNING consensus_state { current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } sequence_number: 108
I20250626 19:58:41.920569 6926 catalog_manager.cc:2747] Deleting table default.table_daf65ec234534a069c2d1ad34f13952a [id=ee116b84a24e49228c7496e95eaf11bb] in response to Hive Metastore notification log event 29
I20250626 19:58:41.921612 8055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 194, Last known committed idx: 193, Time since last communication: 0.000s
I20250626 19:58:41.922616 6717 master_service.cc:948] ReplaceTablet: received request to replace tablet 089e49e29b2b4f3dbea8a4597d9e36a2 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:41.923974 6717 master_service.cc:948] ReplaceTablet: received request to replace tablet 60a00cf0b1854a20b54c4ae8a59ede29 from {username='slave'} at 127.0.0.1:42822
I20250626 19:58:41.941269 8052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6610553ba0df459085f7373935774377. Latest consensus state: current_term: 2 leader_uuid: "6610553ba0df459085f7373935774377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:41.942004 8052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: This master's current role is: FOLLOWER
W20250626 19:58:41.947801 7418 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.4.46.190:39311)
I20250626 19:58:41.954583 6868 master_service.cc:948] ReplaceTablet: received request to replace tablet 089e49e29b2b4f3dbea8a4597d9e36a2 from {username='slave'} at 127.0.0.1:56016
I20250626 19:58:41.973331 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d677beae63a49ecbd11ed53e9e421fc" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:41.982178 7980 leader_election.cc:290] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Requested vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:41.997900 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "196180f1e78a4d5ca8aca8d7bb666d42" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:41.998576 7360 raft_consensus.cc:3058] T 196180f1e78a4d5ca8aca8d7bb666d42 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:42.003412 6710 catalog_manager.cc:3460] renamed table default.table_6a693c4bae214626a4050c7a94276f98 [id=7c0b8a662d1442b5b758ace1e136d79a] in HMS: new name default.table_30a8077ecd80407991c244d577d16cad
I20250626 19:58:41.997205 7009 leader_election.cc:304] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
W20250626 19:58:42.016275 7006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:40373, user_credentials={real_user=slave}} blocked reactor thread for 54999.4us
I20250626 19:58:42.017557 7569 raft_consensus.cc:357] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:42.018203 7569 raft_consensus.cc:383] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:42.017899 7214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "196180f1e78a4d5ca8aca8d7bb666d42" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:42.018445 7569 raft_consensus.cc:738] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:42.018716 7214 raft_consensus.cc:3058] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:42.019078 7569 consensus_queue.cc:260] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:42.021080 7569 ts_tablet_manager.cc:1428] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.148s user 0.004s sys 0.000s
I20250626 19:58:42.024370 8066 leader_election.cc:290] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:42.028728 7214 raft_consensus.cc:2466] T 196180f1e78a4d5ca8aca8d7bb666d42 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:42.016454 8075 raft_consensus.cc:3058] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:42.042631 7360 raft_consensus.cc:2466] T 196180f1e78a4d5ca8aca8d7bb666d42 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:42.046149 7009 leader_election.cc:304] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
W20250626 19:58:42.053200 6869 catalog_manager.cc:5415] Ignoring report from unknown tablet 089e49e29b2b4f3dbea8a4597d9e36a2
W20250626 19:58:42.058568 6867 catalog_manager.cc:5415] Ignoring report from unknown tablet e267bd75d7d342a3a6c28b17eb37d83e
W20250626 19:58:42.059518 6869 catalog_manager.cc:5415] Ignoring report from unknown tablet ae04997b77924e8ba479c5c34ac822d7
W20250626 19:58:42.060860 6869 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
I20250626 19:58:42.061389 6870 master_service.cc:948] ReplaceTablet: received request to replace tablet 60a00cf0b1854a20b54c4ae8a59ede29 from {username='slave'} at 127.0.0.1:56016
I20250626 19:58:42.066196 8068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6610553ba0df459085f7373935774377. Latest consensus state: current_term: 2 leader_uuid: "6610553ba0df459085f7373935774377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:42.066987 8068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:58:42.069747 8070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6610553ba0df459085f7373935774377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:42.070488 8070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [sys.catalog]: This master's current role is: LEADER
I20250626 19:58:42.073186 8067 raft_consensus.cc:2802] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:42.074784 8067 raft_consensus.cc:695] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Becoming Leader. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Running, Role: LEADER
I20250626 19:58:42.075598 8067 consensus_queue.cc:237] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:42.084547 8075 raft_consensus.cc:2747] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:58:42.084630 6709 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_afaa8d56134e47008a0eb5b4de5c35da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 19:58:42.091480 6715 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_09c2ccbfee7e48e4a354a5138f0855d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:42.100405 7952 consensus_queue.cc:1035] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:42.123773 8083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6610553ba0df459085f7373935774377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:42.126263 8083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [sys.catalog]: This master's current role is: LEADER
I20250626 19:58:42.131004 8068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6610553ba0df459085f7373935774377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:42.131703 8068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:58:42.142753 7406 heartbeater.cc:499] Master 127.4.46.188:38919 was elected leader, sending a full tablet report...
I20250626 19:58:42.163996 8082 consensus_queue.cc:1035] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250626 19:58:42.171228 6711 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_df2c3f4a394c42cc81d626c1e46157f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:42.173655 7261 heartbeater.cc:499] Master 127.4.46.188:38919 was elected leader, sending a full tablet report...
W20250626 19:58:42.175458 6716 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1d890136b93b48cab4d218cbd6a72bba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 19:58:42.178355 6712 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42822 (request call id 110) took 2222 ms. Trace:
I20250626 19:58:42.178813 6712 rpcz_store.cc:276] 0626 19:58:39.955609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:39.958893 (+ 3284us) service_pool.cc:225] Handling call
0626 19:58:40.527987 (+569094us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:42.178149 (+1650162us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":6997,"Hive Metastore.run_cpu_time_us":340,"Hive Metastore.run_wall_time_us":665239}
I20250626 19:58:42.183140 6718 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42822 (request call id 111) took 2219 ms. Trace:
I20250626 19:58:42.183519 6718 rpcz_store.cc:276] 0626 19:58:39.963892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:39.964043 (+ 151us) service_pool.cc:225] Handling call
0626 19:58:40.528845 (+564802us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:42.182965 (+1654120us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":671579,"Hive Metastore.run_cpu_time_us":317,"Hive Metastore.run_wall_time_us":591212}
I20250626 19:58:42.185995 6718 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:54470 (request call id 1) took 1298 ms. Trace:
I20250626 19:58:42.189419 6710 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42822 (request call id 113) took 2198 ms. Trace:
I20250626 19:58:42.190336 6869 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:42.191800 8052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6610553ba0df459085f7373935774377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:42.192646 8052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:58:42.189759 6710 rpcz_store.cc:276] 0626 19:58:39.990527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:39.992006 (+ 1479us) service_pool.cc:225] Handling call
0626 19:58:40.529687 (+537681us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:42.187029 (+1657342us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:42.188144 (+ 1115us) catalog_manager.cc:3683] Apply alter schema
0626 19:58:42.189195 (+ 1051us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":1261856,"Hive Metastore.run_cpu_time_us":1773,"Hive Metastore.run_wall_time_us":210909}
I20250626 19:58:42.197490 6926 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: Sending DeleteTablet for 3 replicas of tablet 60a00cf0b1854a20b54c4ae8a59ede29
I20250626 19:58:42.186445 6718 rpcz_store.cc:276] 0626 19:58:40.888028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:42.184860 (+1296832us) service_pool.cc:225] Handling call
0626 19:58:42.185915 (+ 1055us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 19:58:42.247799 6945 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 19:58:42.273684 7073 raft_consensus.cc:3058] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:42.282878 8086 consensus_queue.cc:1035] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:42.290589 6926 catalog_manager.cc:2747] Deleting table default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a] in response to Hive Metastore notification log event 30
I20250626 19:58:42.316833 6870 master_service.cc:948] ReplaceTablet: received request to replace tablet 9d677beae63a49ecbd11ed53e9e421fc from {username='slave'} at 127.0.0.1:56016
I20250626 19:58:42.342963 7952 consensus_queue.cc:1035] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:42.418531 6867 master_service.cc:948] ReplaceTablet: received request to replace tablet a5646bc4c82d4c23941b27994648ee2e from {username='slave'} at 127.0.0.1:56016
I20250626 19:58:42.444214 6866 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56016:
table { table_name: "default.Table_d04e336d056d484ab051754a2b012d9b" } modify_external_catalogs: true
I20250626 19:58:42.445815 6866 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56016:
table { table_name: "default.Table_d04e336d056d484ab051754a2b012d9b" } modify_external_catalogs: true
I20250626 19:58:42.451051 8052 raft_consensus.cc:491] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6610553ba0df459085f7373935774377)
I20250626 19:58:42.451596 8052 raft_consensus.cc:513] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:42.455214 8052 leader_election.cc:290] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
I20250626 19:58:42.456775 7814 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "188d02ec81e645229acdc5eed98c9f9f" candidate_term: 3 candidate_status { last_received { term: 2 index: 194 } } ignore_live_leader: false dest_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" is_pre_election: true
I20250626 19:58:42.456933 6865 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56016:
table { table_name: "default.Table_a050ca8b9a9043ee87ca530228790f54" } modify_external_catalogs: true
I20250626 19:58:42.457628 6865 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56016:
table { table_name: "default.Table_a050ca8b9a9043ee87ca530228790f54" } modify_external_catalogs: true
I20250626 19:58:42.461263 6864 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56016:
table { table_name: "default.Table_63a5ae538eef4f739a22697dfcfb56c7" } new_table_name: "default.Table_be3ed77e691a4f68b7ddb861a1809419" modify_external_catalogs: true
I20250626 19:58:42.481617 7983 consensus_queue.cc:1035] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:42.509902 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 60a00cf0b1854a20b54c4ae8a59ede29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:41 UTC) from {username='slave'} at 127.0.0.1:57994
I20250626 19:58:42.514561 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet 60a00cf0b1854a20b54c4ae8a59ede29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:41 UTC) from {username='slave'} at 127.0.0.1:52438
I20250626 19:58:42.532343 8101 tablet_replica.cc:331] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:42.533968 8101 raft_consensus.cc:2241] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:42.534621 8101 raft_consensus.cc:2270] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:42.553009 8101 ts_tablet_manager.cc:1905] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:58:42.570454 6868 catalog_manager.cc:5415] Ignoring report from unknown tablet e267bd75d7d342a3a6c28b17eb37d83e
W20250626 19:58:42.570962 6868 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
W20250626 19:58:42.571271 6868 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
I20250626 19:58:42.572057 6870 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: Sending DeleteTablet for 3 replicas of tablet 9d677beae63a49ecbd11ed53e9e421fc
I20250626 19:58:42.574728 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d677beae63a49ecbd11ed53e9e421fc with delete_type TABLET_DATA_DELETED (replaced by tablet 08c4b36ad87c47f2a72a3118c3a66fd1) from {username='slave'} at 127.0.0.1:52438
I20250626 19:58:42.576946 6870 catalog_manager.cc:6652] ReplaceTablet: tablet 9d677beae63a49ecbd11ed53e9e421fc deleted and replaced by tablet 08c4b36ad87c47f2a72a3118c3a66fd1
I20250626 19:58:42.576992 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d677beae63a49ecbd11ed53e9e421fc with delete_type TABLET_DATA_DELETED (replaced by tablet 08c4b36ad87c47f2a72a3118c3a66fd1) from {username='slave'} at 127.0.0.1:57994
I20250626 19:58:42.578991 6867 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: Sending DeleteTablet for 3 replicas of tablet a5646bc4c82d4c23941b27994648ee2e
I20250626 19:58:42.581341 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet a5646bc4c82d4c23941b27994648ee2e with delete_type TABLET_DATA_DELETED (replaced by tablet cbac3cf560fb4f41ac7ddac25bcb9e5b) from {username='slave'} at 127.0.0.1:57994
I20250626 19:58:42.583168 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet a5646bc4c82d4c23941b27994648ee2e with delete_type TABLET_DATA_DELETED (replaced by tablet cbac3cf560fb4f41ac7ddac25bcb9e5b) from {username='slave'} at 127.0.0.1:52438
I20250626 19:58:42.583964 6867 catalog_manager.cc:6652] ReplaceTablet: tablet a5646bc4c82d4c23941b27994648ee2e deleted and replaced by tablet cbac3cf560fb4f41ac7ddac25bcb9e5b
I20250626 19:58:42.584062 8101 ts_tablet_manager.cc:1918] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250626 19:58:42.584573 8101 log.cc:1199] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/60a00cf0b1854a20b54c4ae8a59ede29
I20250626 19:58:42.585050 6879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "188d02ec81e645229acdc5eed98c9f9f" candidate_term: 3 candidate_status { last_received { term: 2 index: 194 } } ignore_live_leader: false dest_uuid: "6610553ba0df459085f7373935774377" is_pre_election: true
I20250626 19:58:42.586112 8101 ts_tablet_manager.cc:1939] T 60a00cf0b1854a20b54c4ae8a59ede29 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:42.586437 6705 leader_election.cc:304] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 188d02ec81e645229acdc5eed98c9f9f; no voters: 6610553ba0df459085f7373935774377, 8ae38ec946e14198ae4adc8dde9d66e8
I20250626 19:58:42.587471 8052 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250626 19:58:42.592157 6863 catalog_manager.cc:5415] Ignoring report from unknown tablet 089e49e29b2b4f3dbea8a4597d9e36a2
W20250626 19:58:42.593564 6863 catalog_manager.cc:5415] Ignoring report from unknown tablet e267bd75d7d342a3a6c28b17eb37d83e
W20250626 19:58:42.593858 6863 catalog_manager.cc:5415] Ignoring report from unknown tablet 60a00cf0b1854a20b54c4ae8a59ede29
W20250626 19:58:42.594061 6863 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
I20250626 19:58:42.594805 8101 tablet_replica.cc:331] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:42.595703 8101 raft_consensus.cc:2241] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:42.599797 8101 raft_consensus.cc:2270] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:42.605099 8101 ts_tablet_manager.cc:1905] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:42.613613 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 60a00cf0b1854a20b54c4ae8a59ede29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:41 UTC) from {username='slave'} at 127.0.0.1:50600
I20250626 19:58:42.614363 7053 tablet_service.cc:1515] Processing DeleteTablet for tablet a5646bc4c82d4c23941b27994648ee2e with delete_type TABLET_DATA_DELETED (replaced by tablet cbac3cf560fb4f41ac7ddac25bcb9e5b) from {username='slave'} at 127.0.0.1:50600
I20250626 19:58:42.614363 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d677beae63a49ecbd11ed53e9e421fc with delete_type TABLET_DATA_DELETED (replaced by tablet 08c4b36ad87c47f2a72a3118c3a66fd1) from {username='slave'} at 127.0.0.1:50600
I20250626 19:58:42.592607 6857 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 60a00cf0b1854a20b54c4ae8a59ede29 (table default.table_daf65ec234534a069c2d1ad34f13952a [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:42.620038 8067 raft_consensus.cc:491] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:42.620504 8067 raft_consensus.cc:513] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:42.624449 7359 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ced063ac30b4168b4af4cf2ba239e25" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:42.625727 8106 tablet_replica.cc:331] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:42.626982 7214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ced063ac30b4168b4af4cf2ba239e25" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:42.627905 7214 raft_consensus.cc:2391] T 2ced063ac30b4168b4af4cf2ba239e25 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbe151be01e24c82854d9c8c79c797ca in current term 1: Already voted for candidate 297d2c913667408bae82668a557e06c7 in this term.
I20250626 19:58:42.629552 7006 leader_election.cc:304] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:42.629973 8067 leader_election.cc:290] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:42.630578 8067 raft_consensus.cc:3058] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:42.631397 6863 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
I20250626 19:58:42.631686 8106 raft_consensus.cc:2241] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:42.633147 8106 raft_consensus.cc:2270] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:42.642825 8102 tablet_replica.cc:331] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:42.643900 8102 raft_consensus.cc:2241] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:42.644634 8102 raft_consensus.cc:2270] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:42.644658 8106 ts_tablet_manager.cc:1905] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:42.655866 8011 consensus_queue.cc:1035] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:42.681088 8101 ts_tablet_manager.cc:1918] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:42.681497 8101 log.cc:1199] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/9d677beae63a49ecbd11ed53e9e421fc
I20250626 19:58:42.678541 8067 raft_consensus.cc:2747] T 2ced063ac30b4168b4af4cf2ba239e25 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 19:58:42.683488 8101 ts_tablet_manager.cc:1939] T 9d677beae63a49ecbd11ed53e9e421fc P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:42.687093 6857 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]) successfully deleted
I20250626 19:58:42.691666 6709 catalog_manager.cc:2149] created HMS catalog entry for table [id=dac9e0aba8c943fabd12ce48a2dbd287]
W20250626 19:58:42.693562 6709 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 188d02ec81e645229acdc5eed98c9f9f is not leader of this config: current role FOLLOWER
I20250626 19:58:42.686148 8101 tablet_replica.cc:331] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:42.703226 8101 raft_consensus.cc:2241] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:42.703800 8101 raft_consensus.cc:2270] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:42.706216 8101 ts_tablet_manager.cc:1905] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:42.713515 8106 ts_tablet_manager.cc:1918] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250626 19:58:42.721822 8106 log.cc:1199] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/60a00cf0b1854a20b54c4ae8a59ede29
I20250626 19:58:42.722808 8106 ts_tablet_manager.cc:1939] T 60a00cf0b1854a20b54c4ae8a59ede29 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:42.725839 6855 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 60a00cf0b1854a20b54c4ae8a59ede29 (table default.table_daf65ec234534a069c2d1ad34f13952a [id=ee116b84a24e49228c7496e95eaf11bb]) successfully deleted
I20250626 19:58:42.726876 8106 tablet_replica.cc:331] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:42.727824 8106 raft_consensus.cc:2241] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:42.728392 8106 raft_consensus.cc:2270] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:42.731266 8106 ts_tablet_manager.cc:1905] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:42.741315 6868 catalog_manager.cc:5582] T 0487e1d34dd74d70a5b5c8ac8cc0f636 P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:42.746582 8102 ts_tablet_manager.cc:1905] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:42.761803 8086 consensus_queue.cc:1035] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:42.763132 7952 consensus_queue.cc:1035] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:42.768814 8101 ts_tablet_manager.cc:1918] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:42.769232 8101 log.cc:1199] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/a5646bc4c82d4c23941b27994648ee2e
I20250626 19:58:42.769999 8101 ts_tablet_manager.cc:1939] T a5646bc4c82d4c23941b27994648ee2e P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:42.769418 6868 catalog_manager.cc:5582] T 2ced063ac30b4168b4af4cf2ba239e25 P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:42.788671 6857 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet a5646bc4c82d4c23941b27994648ee2e (table default.table_63a5ae538eef4f739a22697dfcfb56c7 [id=22351693a59e46a9b9491eb79f1d0fe0]) successfully deleted
I20250626 19:58:42.807775 7360 raft_consensus.cc:1273] T e267bd75d7d342a3a6c28b17eb37d83e P 297d2c913667408bae82668a557e06c7 [term 2 FOLLOWER]: Refusing update from remote peer 315d304854c541bf8b5e6b1eea712d4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250626 19:58:42.809221 8011 consensus_queue.cc:1035] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:58:42.831440 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 1: this message will repeat every 5th retry.
I20250626 19:58:42.842905 4282 master-stress-test.cc:577] Killing test thread 1/10
I20250626 19:58:42.861490 6926 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: Sending DeleteTablet for 3 replicas of tablet 83493d647cc8464e8e650bbca7c6072c
I20250626 19:58:42.863534 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 83493d647cc8464e8e650bbca7c6072c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:42 UTC) from {username='slave'} at 127.0.0.1:50600
I20250626 19:58:42.864487 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet 83493d647cc8464e8e650bbca7c6072c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:42 UTC) from {username='slave'} at 127.0.0.1:52438
I20250626 19:58:42.867588 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet 83493d647cc8464e8e650bbca7c6072c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:42 UTC) from {username='slave'} at 127.0.0.1:57994
I20250626 19:58:42.868300 8101 tablet_replica.cc:331] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:42.868803 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 60a00cf0b1854a20b54c4ae8a59ede29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:41 UTC) from {username='slave'} at 127.0.0.1:50600
I20250626 19:58:42.869140 8101 raft_consensus.cc:2241] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:42.870039 8101 raft_consensus.cc:2270] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 19:58:42.870455 6869 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 9d677beae63a49ecbd11ed53e9e421fc consensus state has no leader
W20250626 19:58:42.871371 6869 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.129:56043 (request call id 173) took 899 ms (client timeout 1000 ms). Trace:
I20250626 19:58:42.872483 8101 ts_tablet_manager.cc:1905] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:58:42.871726 6869 rpcz_store.cc:269] 0626 19:58:41.971522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:41.974786 (+ 3264us) service_pool.cc:225] Handling call
0626 19:58:42.050656 (+ 75870us) spinlock_profiling.cc:243] Waited 72.7 ms on lock 0x7b54000104d4. stack: 00007fcb00675483 00007fcb00675380 00007fcb001f1574 00000000004fa174 00000000004fa109 00007fcb07577a70 00007fcb075fbb1a 00007fcb092a8fac 00007fcb093a17ee 00007fcb070b8624 00007fcb070b85b2 00007fcb070b84e1 00007fcb070b845c 00007fcb070b7712 00007fcb053bbdcc 00007fcb053bb206
0626 19:58:42.871304 (+820648us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":88327,"reported_tablets":18,"spinlock_wait_cycles":167198464}
I20250626 19:58:42.895991 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (DEFAULT_TABLE table=default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:42.896972 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08c4b36ad87c47f2a72a3118c3a66fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:42.903455 8106 ts_tablet_manager.cc:1918] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250626 19:58:42.908605 6856 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9d677beae63a49ecbd11ed53e9e421fc consensus state has no leader
I20250626 19:58:42.916983 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (DEFAULT_TABLE table=default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:42.918025 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08c4b36ad87c47f2a72a3118c3a66fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:42.931161 8101 ts_tablet_manager.cc:1918] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:42.932814 8101 log.cc:1199] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/83493d647cc8464e8e650bbca7c6072c
I20250626 19:58:42.933545 8101 ts_tablet_manager.cc:1939] T 83493d647cc8464e8e650bbca7c6072c P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:42.934031 7193 tablet_service.cc:1468] Processing CreateTablet for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (DEFAULT_TABLE table=default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:42.935104 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08c4b36ad87c47f2a72a3118c3a66fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:42.940026 6869 catalog_manager.cc:5582] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca reported cstate change: term changed from 0 to 1, leader changed from <none> to bbe151be01e24c82854d9c8c79c797ca (127.4.46.129). New cstate: current_term: 1 leader_uuid: "bbe151be01e24c82854d9c8c79c797ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:42.942389 6867 master_service.cc:948] ReplaceTablet: received request to replace tablet 7b9a0f71535a4a65a775f728edbcc636 from {username='slave'} at 127.0.0.1:56016
I20250626 19:58:42.945900 6870 master_service.cc:948] ReplaceTablet: received request to replace tablet f9adef69fe7442a4836a84934e792c05 from {username='slave'} at 127.0.0.1:56016
I20250626 19:58:42.972060 7194 tablet_service.cc:1468] Processing CreateTablet for tablet cbac3cf560fb4f41ac7ddac25bcb9e5b (DEFAULT_TABLE table=default.table_63a5ae538eef4f739a22697dfcfb56c7 [id=22351693a59e46a9b9491eb79f1d0fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:42.973155 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbac3cf560fb4f41ac7ddac25bcb9e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:42.968840 7051 tablet_service.cc:1468] Processing CreateTablet for tablet cbac3cf560fb4f41ac7ddac25bcb9e5b (DEFAULT_TABLE table=default.table_63a5ae538eef4f739a22697dfcfb56c7 [id=22351693a59e46a9b9491eb79f1d0fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:42.992094 7051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbac3cf560fb4f41ac7ddac25bcb9e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:43.009727 8106 log.cc:1199] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/a5646bc4c82d4c23941b27994648ee2e
I20250626 19:58:43.010720 8106 ts_tablet_manager.cc:1939] T a5646bc4c82d4c23941b27994648ee2e P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:43.013243 6855 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet a5646bc4c82d4c23941b27994648ee2e (table default.table_63a5ae538eef4f739a22697dfcfb56c7 [id=22351693a59e46a9b9491eb79f1d0fe0]) successfully deleted
I20250626 19:58:43.014166 8106 tablet_replica.cc:331] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:43.015059 8106 raft_consensus.cc:2241] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:43.015589 8106 raft_consensus.cc:2270] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:43.018391 8106 ts_tablet_manager.cc:1905] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:43.037657 8115 tablet_bootstrap.cc:492] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:43.039443 8106 ts_tablet_manager.cc:1918] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:43.040601 8106 log.cc:1199] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/9d677beae63a49ecbd11ed53e9e421fc
I20250626 19:58:43.041432 8106 ts_tablet_manager.cc:1939] T 9d677beae63a49ecbd11ed53e9e421fc P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:43.044472 6855 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]) successfully deleted
I20250626 19:58:43.045419 8106 tablet_replica.cc:331] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:43.047117 8106 raft_consensus.cc:2241] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:43.047689 8106 raft_consensus.cc:2270] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:43.049611 8106 ts_tablet_manager.cc:1905] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:43.075805 8115 tablet_bootstrap.cc:654] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:43.075711 8106 ts_tablet_manager.cc:1918] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:43.076249 8106 log.cc:1199] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/83493d647cc8464e8e650bbca7c6072c
I20250626 19:58:43.077181 8106 ts_tablet_manager.cc:1939] T 83493d647cc8464e8e650bbca7c6072c P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:43.083235 6855 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 83493d647cc8464e8e650bbca7c6072c (table default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]) successfully deleted
I20250626 19:58:42.952512 6857 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 83493d647cc8464e8e650bbca7c6072c (table default.table_5283819977d9487a9d1aaae326249a46 [id=2fd44676faf246d285e450e5918c169a]) successfully deleted
W20250626 19:58:43.087468 6855 catalog_manager.cc:4895] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 60a00cf0b1854a20b54c4ae8a59ede29 because the tablet was not found. No further retry: Not found: Tablet not found: 60a00cf0b1854a20b54c4ae8a59ede29
W20250626 19:58:43.087628 6857 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.46.131:33703, user_credentials={real_user=slave}} blocked reactor thread for 135181us
I20250626 19:58:43.089745 7340 tablet_service.cc:1468] Processing CreateTablet for tablet cbac3cf560fb4f41ac7ddac25bcb9e5b (DEFAULT_TABLE table=default.table_63a5ae538eef4f739a22697dfcfb56c7 [id=22351693a59e46a9b9491eb79f1d0fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:43.090859 7340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbac3cf560fb4f41ac7ddac25bcb9e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:43.090921 6867 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: Sending DeleteTablet for 3 replicas of tablet 7b9a0f71535a4a65a775f728edbcc636
I20250626 19:58:43.094282 6870 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: Sending DeleteTablet for 3 replicas of tablet f9adef69fe7442a4836a84934e792c05
I20250626 19:58:43.095719 6870 catalog_manager.cc:6652] ReplaceTablet: tablet f9adef69fe7442a4836a84934e792c05 deleted and replaced by tablet 0084d8b5ae1548af8c943f8ace9bceb2
I20250626 19:58:43.096091 7051 tablet_service.cc:1515] Processing DeleteTablet for tablet 7b9a0f71535a4a65a775f728edbcc636 with delete_type TABLET_DATA_DELETED (replaced by tablet 0aaf2ab9ed7840898ca54cca6983eb82) from {username='slave'} at 127.0.0.1:50600
I20250626 19:58:43.096525 6867 catalog_manager.cc:6652] ReplaceTablet: tablet 7b9a0f71535a4a65a775f728edbcc636 deleted and replaced by tablet 0aaf2ab9ed7840898ca54cca6983eb82
I20250626 19:58:43.096768 7053 tablet_service.cc:1515] Processing DeleteTablet for tablet f9adef69fe7442a4836a84934e792c05 with delete_type TABLET_DATA_DELETED (replaced by tablet 0084d8b5ae1548af8c943f8ace9bceb2) from {username='slave'} at 127.0.0.1:50600
I20250626 19:58:43.099124 7193 tablet_service.cc:1515] Processing DeleteTablet for tablet f9adef69fe7442a4836a84934e792c05 with delete_type TABLET_DATA_DELETED (replaced by tablet 0084d8b5ae1548af8c943f8ace9bceb2) from {username='slave'} at 127.0.0.1:52438
I20250626 19:58:43.101379 7341 tablet_service.cc:1515] Processing DeleteTablet for tablet 7b9a0f71535a4a65a775f728edbcc636 with delete_type TABLET_DATA_DELETED (replaced by tablet 0aaf2ab9ed7840898ca54cca6983eb82) from {username='slave'} at 127.0.0.1:57994
I20250626 19:58:43.101634 7194 tablet_service.cc:1515] Processing DeleteTablet for tablet 7b9a0f71535a4a65a775f728edbcc636 with delete_type TABLET_DATA_DELETED (replaced by tablet 0aaf2ab9ed7840898ca54cca6983eb82) from {username='slave'} at 127.0.0.1:52438
I20250626 19:58:43.102077 8101 tablet_replica.cc:331] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:43.102909 8101 raft_consensus.cc:2241] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:43.103767 8101 raft_consensus.cc:2270] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:43.105664 8101 ts_tablet_manager.cc:1905] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:43.106498 8116 tablet_bootstrap.cc:492] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:43.101379 7337 tablet_service.cc:1515] Processing DeleteTablet for tablet f9adef69fe7442a4836a84934e792c05 with delete_type TABLET_DATA_DELETED (replaced by tablet 0084d8b5ae1548af8c943f8ace9bceb2) from {username='slave'} at 127.0.0.1:57994
I20250626 19:58:43.108469 8106 tablet_replica.cc:331] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:43.108742 8067 consensus_queue.cc:1035] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:43.110055 8106 raft_consensus.cc:2241] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:43.110579 8106 raft_consensus.cc:2270] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:43.127769 8114 tablet_bootstrap.cc:492] T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:43.131846 8106 ts_tablet_manager.cc:1905] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:43.145562 8115 tablet_bootstrap.cc:492] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:43.146150 8115 ts_tablet_manager.cc:1397] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.109s user 0.012s sys 0.004s
I20250626 19:58:43.148792 8115 raft_consensus.cc:357] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.149410 8115 raft_consensus.cc:383] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:43.149693 8115 raft_consensus.cc:738] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:43.150310 8115 consensus_queue.cc:260] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.157902 8115 ts_tablet_manager.cc:1428] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.011s user 0.002s sys 0.003s
I20250626 19:58:43.164837 8115 tablet_bootstrap.cc:492] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:43.176846 8075 consensus_queue.cc:1035] T 196180f1e78a4d5ca8aca8d7bb666d42 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:43.180094 8115 tablet_bootstrap.cc:654] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:43.182039 8101 ts_tablet_manager.cc:1918] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:58:43.182474 8101 log.cc:1199] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/7b9a0f71535a4a65a775f728edbcc636
I20250626 19:58:43.183320 8101 ts_tablet_manager.cc:1939] T 7b9a0f71535a4a65a775f728edbcc636 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:43.185853 8101 tablet_replica.cc:331] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7: stopping tablet replica
I20250626 19:58:43.189510 6857 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet 7b9a0f71535a4a65a775f728edbcc636 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:43.190106 8106 ts_tablet_manager.cc:1918] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:58:43.186671 8101 raft_consensus.cc:2241] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:43.195026 8101 raft_consensus.cc:2270] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:43.196394 7340 tablet_service.cc:1515] Processing DeleteTablet for tablet 60a00cf0b1854a20b54c4ae8a59ede29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:41 UTC) from {username='slave'} at 127.0.0.1:57994
I20250626 19:58:43.196992 7337 tablet_service.cc:1515] Processing DeleteTablet for tablet 83493d647cc8464e8e650bbca7c6072c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 19:58:42 UTC) from {username='slave'} at 127.0.0.1:57994
I20250626 19:58:43.197577 7341 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d677beae63a49ecbd11ed53e9e421fc with delete_type TABLET_DATA_DELETED (replaced by tablet 08c4b36ad87c47f2a72a3118c3a66fd1) from {username='slave'} at 127.0.0.1:57994
I20250626 19:58:43.198186 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet a5646bc4c82d4c23941b27994648ee2e with delete_type TABLET_DATA_DELETED (replaced by tablet cbac3cf560fb4f41ac7ddac25bcb9e5b) from {username='slave'} at 127.0.0.1:57994
I20250626 19:58:43.199795 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d677beae63a49ecbd11ed53e9e421fc with delete_type TABLET_DATA_DELETED (replaced by tablet 08c4b36ad87c47f2a72a3118c3a66fd1) from {username='slave'} at 127.0.0.1:50600
I20250626 19:58:43.200802 7050 tablet_service.cc:1515] Processing DeleteTablet for tablet a5646bc4c82d4c23941b27994648ee2e with delete_type TABLET_DATA_DELETED (replaced by tablet cbac3cf560fb4f41ac7ddac25bcb9e5b) from {username='slave'} at 127.0.0.1:50600
I20250626 19:58:43.190505 8106 log.cc:1199] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/7b9a0f71535a4a65a775f728edbcc636
I20250626 19:58:43.206166 8106 ts_tablet_manager.cc:1939] T 7b9a0f71535a4a65a775f728edbcc636 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:43.208803 8106 tablet_replica.cc:331] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca: stopping tablet replica
I20250626 19:58:43.208858 6855 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet 7b9a0f71535a4a65a775f728edbcc636 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:43.209710 8106 raft_consensus.cc:2241] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:43.210083 8101 ts_tablet_manager.cc:1905] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:43.216996 8106 raft_consensus.cc:2270] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:43.218926 8106 ts_tablet_manager.cc:1905] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:43.233799 8102 ts_tablet_manager.cc:1918] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250626 19:58:43.234457 8102 log.cc:1199] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/60a00cf0b1854a20b54c4ae8a59ede29
I20250626 19:58:43.257541 8114 tablet_bootstrap.cc:654] T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:43.261632 6714 catalog_manager.cc:3460] renamed table default.table_dc0e9650e86e4c60aec4f82f9b82f959 [id=d1d807f596fd4519b4d74850a971205a] in HMS: new name default.table_fd11e459e1d9471fae895dcdf01e89fa
I20250626 19:58:43.266631 6714 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42822 (request call id 114) took 2966 ms. Trace:
I20250626 19:58:43.267340 6714 rpcz_store.cc:276] 0626 19:58:40.299863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:40.304991 (+ 5128us) service_pool.cc:225] Handling call
0626 19:58:42.088896 (+1783905us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:43.263134 (+1174238us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 19:58:43.265632 (+ 2498us) catalog_manager.cc:3683] Apply alter schema
0626 19:58:43.266555 (+ 923us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":601303,"Hive Metastore.run_cpu_time_us":1836,"Hive Metastore.run_wall_time_us":569548}
I20250626 19:58:43.275398 7406 heartbeater.cc:499] Master 127.4.46.188:38919 was elected leader, sending a full tablet report...
I20250626 19:58:43.275614 8101 ts_tablet_manager.cc:1918] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:43.275945 8101 log.cc:1199] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-2/wal/wals/f9adef69fe7442a4836a84934e792c05
I20250626 19:58:43.276732 8101 ts_tablet_manager.cc:1939] T f9adef69fe7442a4836a84934e792c05 P 297d2c913667408bae82668a557e06c7: Deleting consensus metadata
I20250626 19:58:43.277981 6857 catalog_manager.cc:4928] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): tablet f9adef69fe7442a4836a84934e792c05 (table default.table_a43e8ff4d1844c0b9dddf28fc310fd8f [id=ef672ba275b54933ae739bf9e560ae68]) successfully deleted
I20250626 19:58:43.283897 8106 ts_tablet_manager.cc:1918] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250626 19:58:43.286265 6857 catalog_manager.cc:4895] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 83493d647cc8464e8e650bbca7c6072c because the tablet was not found. No further retry: Not found: Tablet not found: 83493d647cc8464e8e650bbca7c6072c
W20250626 19:58:43.287070 6857 catalog_manager.cc:4895] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet 9d677beae63a49ecbd11ed53e9e421fc because the tablet was not found. No further retry: Not found: Tablet not found: 9d677beae63a49ecbd11ed53e9e421fc
I20250626 19:58:43.288247 8116 tablet_bootstrap.cc:700] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7: Time spent opening tablet: real 0.143s user 0.002s sys 0.000s
W20250626 19:58:43.288174 6857 catalog_manager.cc:4895] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet a5646bc4c82d4c23941b27994648ee2e because the tablet was not found. No further retry: Not found: Tablet not found: a5646bc4c82d4c23941b27994648ee2e
I20250626 19:58:43.288898 8116 tablet_bootstrap.cc:654] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:43.296919 8106 log.cc:1199] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-0/wal/wals/f9adef69fe7442a4836a84934e792c05
I20250626 19:58:43.297973 8106 ts_tablet_manager.cc:1939] T f9adef69fe7442a4836a84934e792c05 P bbe151be01e24c82854d9c8c79c797ca: Deleting consensus metadata
I20250626 19:58:43.300972 6855 catalog_manager.cc:4928] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): tablet f9adef69fe7442a4836a84934e792c05 (table default.table_a43e8ff4d1844c0b9dddf28fc310fd8f [id=ef672ba275b54933ae739bf9e560ae68]) successfully deleted
W20250626 19:58:43.302134 6855 catalog_manager.cc:4895] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet 9d677beae63a49ecbd11ed53e9e421fc because the tablet was not found. No further retry: Not found: Tablet not found: 9d677beae63a49ecbd11ed53e9e421fc
W20250626 19:58:43.302706 6855 catalog_manager.cc:4895] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): delete failed for tablet a5646bc4c82d4c23941b27994648ee2e because the tablet was not found. No further retry: Not found: Tablet not found: a5646bc4c82d4c23941b27994648ee2e
W20250626 19:58:43.288537 7006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.130:40373, user_credentials={real_user=slave}} blocked reactor thread for 113173us
I20250626 19:58:43.325326 8102 ts_tablet_manager.cc:1939] T 60a00cf0b1854a20b54c4ae8a59ede29 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
W20250626 19:58:43.338743 8126 log.cc:927] Time spent T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20250626 19:58:43.361786 8134 tablet.cc:1722] T c7159554b45442b5aa5069d7a47d1f8e P 315d304854c541bf8b5e6b1eea712d4a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:58:43.362273 8115 tablet_bootstrap.cc:492] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:43.363138 8115 ts_tablet_manager.cc:1397] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.198s user 0.039s sys 0.036s
I20250626 19:58:43.364377 8128 tablet.cc:1722] T c7159554b45442b5aa5069d7a47d1f8e P 297d2c913667408bae82668a557e06c7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:58:43.365692 8115 raft_consensus.cc:357] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.366454 8115 raft_consensus.cc:383] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:43.366694 8115 raft_consensus.cc:738] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:43.367384 8115 consensus_queue.cc:260] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.368255 8114 tablet_bootstrap.cc:492] T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:43.367761 8129 tablet.cc:1722] T c7159554b45442b5aa5069d7a47d1f8e P bbe151be01e24c82854d9c8c79c797ca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:58:43.368680 8114 ts_tablet_manager.cc:1397] T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.241s user 0.005s sys 0.005s
I20250626 19:58:43.369614 8115 ts_tablet_manager.cc:1428] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250626 19:58:43.371858 8102 tablet_replica.cc:331] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:43.372731 8102 raft_consensus.cc:2241] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:43.373205 8102 raft_consensus.cc:2270] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 19:58:43.379119 6868 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.46.131:54713 (request call id 177) took 1064 ms (client timeout 1000 ms). Trace:
W20250626 19:58:43.379411 6868 rpcz_store.cc:269] 0626 19:58:42.314579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:42.314839 (+ 260us) service_pool.cc:225] Handling call
0626 19:58:43.379056 (+1064217us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:42.773722 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 19:58:42.774130 (+ 408us) write_op.cc:432] Acquiring schema lock in shared mode
0626 19:58:42.774156 (+ 26us) write_op.cc:435] Acquired schema lock
0626 19:58:42.774171 (+ 15us) tablet.cc:660] Decoding operations
0626 19:58:42.774474 (+ 303us) write_op.cc:620] Acquiring the partition lock for write op
0626 19:58:42.774515 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0626 19:58:42.774606 (+ 91us) tablet.cc:683] Acquiring locks for 2 operations
0626 19:58:42.775002 (+ 396us) tablet.cc:699] Row locks acquired
0626 19:58:42.775028 (+ 26us) write_op.cc:260] PREPARE: finished
0626 19:58:42.775154 (+ 126us) write_op.cc:270] Start()
0626 19:58:42.775244 (+ 90us) write_op.cc:276] Timestamp: P: 1750967922775132 usec, L: 0
0626 19:58:42.775269 (+ 25us) op_driver.cc:348] REPLICATION: starting
0626 19:58:42.775820 (+ 551us) log.cc:844] Serialized 907 byte log entry
0626 19:58:42.789062 (+ 13242us) op_driver.cc:464] REPLICATION: finished
0626 19:58:42.789435 (+ 373us) write_op.cc:301] APPLY: starting
0626 19:58:42.789492 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
0626 19:58:42.789594 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0626 19:58:42.789678 (+ 84us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 19:58:42.789915 (+ 237us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 19:58:42.790048 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0626 19:58:42.790088 (+ 40us) write_op.cc:312] APPLY: finished
0626 19:58:42.790258 (+ 170us) log.cc:844] Serialized 29 byte log entry
0626 19:58:42.790406 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:42.790471 (+ 65us) write_op.cc:454] Released schema lock
0626 19:58:42.790510 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0626 19:58:42.790725 (+ 215us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:42.790767 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":59731,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":207,"apply.run_cpu_time_us":1264,"apply.run_wall_time_us":1264,"num_ops":2,"prepare.queue_time_us":465,"prepare.run_cpu_time_us":2432,"prepare.run_wall_time_us":2431,"raft.queue_time_us":3317,"raft.run_cpu_time_us":1352,"raft.run_wall_time_us":1352,"replication_time_us":13456}]]}
W20250626 19:58:43.384542 6867 catalog_manager.cc:5415] Ignoring report from unknown tablet e267bd75d7d342a3a6c28b17eb37d83e
W20250626 19:58:43.384791 6867 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
W20250626 19:58:43.385185 6867 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
W20250626 19:58:43.390930 6863 catalog_manager.cc:5415] Ignoring report from unknown tablet a5646bc4c82d4c23941b27994648ee2e
W20250626 19:58:43.391172 6863 catalog_manager.cc:5415] Ignoring report from unknown tablet 089e49e29b2b4f3dbea8a4597d9e36a2
W20250626 19:58:43.391579 6863 catalog_manager.cc:5415] Ignoring report from unknown tablet e267bd75d7d342a3a6c28b17eb37d83e
W20250626 19:58:43.391772 6863 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
W20250626 19:58:43.391925 6863 catalog_manager.cc:5415] Ignoring report from unknown tablet 9d677beae63a49ecbd11ed53e9e421fc
I20250626 19:58:43.393568 8116 tablet_bootstrap.cc:492] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:43.394035 8116 ts_tablet_manager.cc:1397] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.288s user 0.011s sys 0.000s
I20250626 19:58:43.396716 8116 raft_consensus.cc:357] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.397342 8116 raft_consensus.cc:383] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:43.397620 8116 raft_consensus.cc:738] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:43.432416 8102 ts_tablet_manager.cc:1905] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:43.435920 8114 raft_consensus.cc:357] T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.436622 8114 raft_consensus.cc:383] T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:43.436935 8114 raft_consensus.cc:738] T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:43.437577 8114 consensus_queue.cc:260] T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.444147 6867 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:43.398227 8116 consensus_queue.cc:260] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.448565 8116 ts_tablet_manager.cc:1428] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.054s user 0.006s sys 0.000s
I20250626 19:58:43.449489 8116 tablet_bootstrap.cc:492] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:43.456934 8114 ts_tablet_manager.cc:1428] T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.088s user 0.002s sys 0.005s
I20250626 19:58:43.458336 8011 raft_consensus.cc:491] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:43.458817 8011 raft_consensus.cc:513] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.462339 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "08c4b36ad87c47f2a72a3118c3a66fd1" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:43.462168 8011 leader_election.cc:290] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:43.462944 7360 raft_consensus.cc:2466] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:43.464597 7152 leader_election.cc:304] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:43.474689 8011 raft_consensus.cc:2802] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:43.475126 8011 raft_consensus.cc:491] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:43.475456 8011 raft_consensus.cc:3058] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:43.495769 8116 tablet_bootstrap.cc:654] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:43.498890 6867 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet c7159554b45442b5aa5069d7a47d1f8e (table default.table_30a8077ecd80407991c244d577d16cad [id=7c0b8a662d1442b5b758ace1e136d79a]). Expected version 1 got 0
I20250626 19:58:43.503850 8011 raft_consensus.cc:513] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.508132 6863 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
W20250626 19:58:43.509308 6867 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 08c4b36ad87c47f2a72a3118c3a66fd1 consensus state has no leader
I20250626 19:58:43.511571 7339 tablet_service.cc:1515] Processing DeleteTablet for tablet f9adef69fe7442a4836a84934e792c05 with delete_type TABLET_DATA_DELETED (replaced by tablet 0084d8b5ae1548af8c943f8ace9bceb2) from {username='slave'} at 127.0.0.1:57994
W20250626 19:58:43.514585 6857 catalog_manager.cc:4895] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): delete failed for tablet f9adef69fe7442a4836a84934e792c05 because the tablet was not found. No further retry: Not found: Tablet not found: f9adef69fe7442a4836a84934e792c05
I20250626 19:58:43.518074 7069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "08c4b36ad87c47f2a72a3118c3a66fd1" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:43.518630 7069 raft_consensus.cc:2466] T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:43.519791 7066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "08c4b36ad87c47f2a72a3118c3a66fd1" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:43.520289 7066 raft_consensus.cc:3058] T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:43.522894 8114 tablet_bootstrap.cc:492] T cbac3cf560fb4f41ac7ddac25bcb9e5b P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:43.527235 6869 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:43.533836 7066 raft_consensus.cc:2466] T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:43.534859 7150 leader_election.cc:304] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:43.539178 8011 leader_election.cc:290] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:43.540266 8011 raft_consensus.cc:2802] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:43.540685 8011 raft_consensus.cc:695] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:43.541538 8011 consensus_queue.cc:237] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.561262 8116 tablet_bootstrap.cc:492] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:43.561790 8116 ts_tablet_manager.cc:1397] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.112s user 0.010s sys 0.000s
I20250626 19:58:43.564392 8116 raft_consensus.cc:357] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.573072 8102 ts_tablet_manager.cc:1918] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:43.573540 8102 log.cc:1199] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/9d677beae63a49ecbd11ed53e9e421fc
I20250626 19:58:43.574333 8102 ts_tablet_manager.cc:1939] T 9d677beae63a49ecbd11ed53e9e421fc P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:43.574915 6863 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:43.585759 6869 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: default.table_30a8077ecd80407991c244d577d16cad [id=7c0b8a662d1442b5b758ace1e136d79a] alter complete (version 1)
I20250626 19:58:43.586078 6866 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]
I20250626 19:58:43.586448 8114 tablet_bootstrap.cc:654] T cbac3cf560fb4f41ac7ddac25bcb9e5b P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:43.595218 6869 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:43.607609 8102 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52438 (request call id 1) took 1033 ms. Trace:
I20250626 19:58:43.574165 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "08c4b36ad87c47f2a72a3118c3a66fd1" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:43.609670 8116 raft_consensus.cc:383] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:43.609717 7360 raft_consensus.cc:3058] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:43.610134 8116 raft_consensus.cc:738] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:43.610911 8116 consensus_queue.cc:260] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.616001 7360 raft_consensus.cc:2466] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:43.628031 8116 ts_tablet_manager.cc:1428] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.066s user 0.005s sys 0.000s
I20250626 19:58:43.634860 8102 rpcz_store.cc:276] 0626 19:58:42.574191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:42.574459 (+ 268us) service_pool.cc:225] Handling call
0626 19:58:43.371696 (+797237us) ts_tablet_manager.cc:1145] Deleting tablet 9d677beae63a49ecbd11ed53e9e421fc
0626 19:58:43.371742 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:58:43.469668 (+ 97926us) tablet_metadata.cc:641] Metadata flushed
0626 19:58:43.572830 (+103162us) tablet_metadata.cc:641] Metadata flushed
0626 19:58:43.607531 (+ 34701us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":796485}
I20250626 19:58:43.636626 8102 tablet_replica.cc:331] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:43.637434 6854 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]) successfully deleted
I20250626 19:58:43.637512 8102 raft_consensus.cc:2241] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Raft consensus shutting down.
I20250626 19:58:43.638530 8102 raft_consensus.cc:2270] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 19:58:43.641536 6863 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
W20250626 19:58:43.641922 6863 catalog_manager.cc:5415] Ignoring report from unknown tablet 089e49e29b2b4f3dbea8a4597d9e36a2
I20250626 19:58:43.649165 6863 catalog_manager.cc:5582] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130). New cstate: current_term: 1 leader_uuid: "315d304854c541bf8b5e6b1eea712d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } }
I20250626 19:58:43.650521 6863 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:43.672901 8102 ts_tablet_manager.cc:1905] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:43.675700 8114 tablet_bootstrap.cc:492] T cbac3cf560fb4f41ac7ddac25bcb9e5b P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:43.676182 8114 ts_tablet_manager.cc:1397] T cbac3cf560fb4f41ac7ddac25bcb9e5b P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.157s user 0.011s sys 0.015s
I20250626 19:58:43.679304 8114 raft_consensus.cc:357] T cbac3cf560fb4f41ac7ddac25bcb9e5b P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.680110 8114 raft_consensus.cc:383] T cbac3cf560fb4f41ac7ddac25bcb9e5b P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:43.680435 8114 raft_consensus.cc:738] T cbac3cf560fb4f41ac7ddac25bcb9e5b P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:43.681208 8114 consensus_queue.cc:260] T cbac3cf560fb4f41ac7ddac25bcb9e5b P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.711921 8114 ts_tablet_manager.cc:1428] T cbac3cf560fb4f41ac7ddac25bcb9e5b P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.035s user 0.002s sys 0.004s
W20250626 19:58:43.712813 6863 catalog_manager.cc:5415] Ignoring report from unknown tablet e267bd75d7d342a3a6c28b17eb37d83e
W20250626 19:58:43.713430 6863 catalog_manager.cc:5415] Ignoring report from unknown tablet 089e49e29b2b4f3dbea8a4597d9e36a2
I20250626 19:58:43.725817 7066 raft_consensus.cc:1273] T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Refusing update from remote peer 315d304854c541bf8b5e6b1eea712d4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:58:43.726522 7359 raft_consensus.cc:1273] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Refusing update from remote peer 315d304854c541bf8b5e6b1eea712d4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 19:58:43.728078 8138 consensus_queue.cc:1035] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:43.729328 8138 consensus_queue.cc:1035] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:43.732314 6863 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:43.750082 6867 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:43.755405 8134 tablet.cc:1722] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 315d304854c541bf8b5e6b1eea712d4a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:58:43.762915 6715 catalog_manager.cc:2149] created HMS catalog entry for table [id=bfe0cd9ec0ce4b009fbe0f0f7471c818]
W20250626 19:58:43.765359 6715 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 188d02ec81e645229acdc5eed98c9f9f is not leader of this config: current role FOLLOWER
I20250626 19:58:43.773572 8128 tablet.cc:1722] T 08c4b36ad87c47f2a72a3118c3a66fd1 P 297d2c913667408bae82668a557e06c7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:58:43.766508 6869 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:43.780401 8138 raft_consensus.cc:491] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:43.780869 8138 raft_consensus.cc:513] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.783149 8138 leader_election.cc:290] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:43.783922 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbac3cf560fb4f41ac7ddac25bcb9e5b" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:43.784488 7073 raft_consensus.cc:2466] T cbac3cf560fb4f41ac7ddac25bcb9e5b P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:43.785902 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbac3cf560fb4f41ac7ddac25bcb9e5b" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:43.786420 7360 raft_consensus.cc:2466] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 0.
I20250626 19:58:43.787629 7152 leader_election.cc:304] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:43.797075 8138 raft_consensus.cc:2802] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:43.797451 8138 raft_consensus.cc:491] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:43.797887 8138 raft_consensus.cc:3058] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
W20250626 19:58:43.801534 8146 log.cc:927] Time spent T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250626 19:58:43.809049 8129 tablet.cc:1722] T 08c4b36ad87c47f2a72a3118c3a66fd1 P bbe151be01e24c82854d9c8c79c797ca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:58:43.829524 6863 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]). Expected version 1 got 0
I20250626 19:58:43.849589 8138 raft_consensus.cc:513] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.854171 6867 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377: default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7] alter complete (version 1)
I20250626 19:58:43.869186 7066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbac3cf560fb4f41ac7ddac25bcb9e5b" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:43.869750 7066 raft_consensus.cc:3058] T cbac3cf560fb4f41ac7ddac25bcb9e5b P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:43.876609 7066 raft_consensus.cc:2466] T cbac3cf560fb4f41ac7ddac25bcb9e5b P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:43.877699 7150 leader_election.cc:304] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:43.878432 8011 raft_consensus.cc:2802] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:43.878906 8011 raft_consensus.cc:695] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [term 1 LEADER]: Becoming Leader. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Running, Role: LEADER
I20250626 19:58:43.879601 8011 consensus_queue.cc:237] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:43.891108 7359 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbac3cf560fb4f41ac7ddac25bcb9e5b" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:43.892093 7359 raft_consensus.cc:3058] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:43.904753 8138 leader_election.cc:290] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
I20250626 19:58:43.911569 6865 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]
I20250626 19:58:43.933053 6867 catalog_manager.cc:5582] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130). New cstate: current_term: 1 leader_uuid: "315d304854c541bf8b5e6b1eea712d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: HEALTHY } } }
I20250626 19:58:43.940481 8102 ts_tablet_manager.cc:1918] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:43.910552 7359 raft_consensus.cc:2466] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315d304854c541bf8b5e6b1eea712d4a in term 1.
I20250626 19:58:43.944386 8102 log.cc:1199] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/a5646bc4c82d4c23941b27994648ee2e
I20250626 19:58:43.950584 8102 ts_tablet_manager.cc:1939] T a5646bc4c82d4c23941b27994648ee2e P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:43.967140 8102 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52438 (request call id 2) took 1384 ms. Trace:
I20250626 19:58:43.972012 8102 rpcz_store.cc:276] 0626 19:58:42.582873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:42.583015 (+ 142us) service_pool.cc:225] Handling call
0626 19:58:43.636452 (+1053437us) ts_tablet_manager.cc:1145] Deleting tablet a5646bc4c82d4c23941b27994648ee2e
0626 19:58:43.636498 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:58:43.719507 (+ 83009us) tablet_metadata.cc:641] Metadata flushed
0626 19:58:43.940225 (+220718us) tablet_metadata.cc:641] Metadata flushed
0626 19:58:43.967065 (+ 26840us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1052903}
I20250626 19:58:43.981431 6854 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet a5646bc4c82d4c23941b27994648ee2e (table default.table_63a5ae538eef4f739a22697dfcfb56c7 [id=22351693a59e46a9b9491eb79f1d0fe0]) successfully deleted
I20250626 19:58:43.986388 6711 catalog_manager.cc:2149] created HMS catalog entry for table [id=1e9ad2d55b0548498fb64172b882d9d0]
W20250626 19:58:43.988492 6711 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 188d02ec81e645229acdc5eed98c9f9f is not leader of this config: current role FOLLOWER
I20250626 19:58:43.993408 8102 tablet_replica.cc:331] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:43.997440 8102 raft_consensus.cc:2241] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:43.998049 8102 raft_consensus.cc:2270] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:44.000332 8102 ts_tablet_manager.cc:1905] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 19:58:44.023610 6703 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9d677beae63a49ecbd11ed53e9e421fc consensus state has no leader
W20250626 19:58:44.025662 6856 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9d677beae63a49ecbd11ed53e9e421fc consensus state has no leader
I20250626 19:58:44.063884 8102 ts_tablet_manager.cc:1918] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:44.064500 8102 log.cc:1199] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/83493d647cc8464e8e650bbca7c6072c
I20250626 19:58:44.065274 8102 ts_tablet_manager.cc:1939] T 83493d647cc8464e8e650bbca7c6072c P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:44.067600 8102 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52438 (request call id 3) took 1203 ms. Trace:
I20250626 19:58:44.070848 8102 rpcz_store.cc:276] 0626 19:58:42.864226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:42.864351 (+ 125us) service_pool.cc:225] Handling call
0626 19:58:43.993237 (+1128886us) ts_tablet_manager.cc:1145] Deleting tablet 83493d647cc8464e8e650bbca7c6072c
0626 19:58:43.993281 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:58:44.055994 (+ 62713us) tablet_metadata.cc:641] Metadata flushed
0626 19:58:44.063652 (+ 7658us) tablet_metadata.cc:641] Metadata flushed
0626 19:58:44.067537 (+ 3885us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1128365}
I20250626 19:58:44.074378 8102 tablet_replica.cc:331] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:44.075285 8102 raft_consensus.cc:2241] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:44.078512 8102 raft_consensus.cc:2270] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:44.080675 8102 ts_tablet_manager.cc:1905] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:44.115074 6716 catalog_manager.cc:2149] created HMS catalog entry for table [id=a187bf25ba884923869c7367c1d00dc4]
W20250626 19:58:44.116977 6716 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 188d02ec81e645229acdc5eed98c9f9f is not leader of this config: current role FOLLOWER
I20250626 19:58:44.121388 8102 ts_tablet_manager.cc:1918] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 19:58:44.121894 8102 log.cc:1199] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/f9adef69fe7442a4836a84934e792c05
I20250626 19:58:44.122684 8102 ts_tablet_manager.cc:1939] T f9adef69fe7442a4836a84934e792c05 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:44.124778 8102 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52438 (request call id 6) took 1026 ms. Trace:
I20250626 19:58:44.131489 8102 rpcz_store.cc:276] 0626 19:58:43.098873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:43.098994 (+ 121us) service_pool.cc:225] Handling call
0626 19:58:44.074192 (+975198us) ts_tablet_manager.cc:1145] Deleting tablet f9adef69fe7442a4836a84934e792c05
0626 19:58:44.074241 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:58:44.098072 (+ 23831us) tablet_metadata.cc:641] Metadata flushed
0626 19:58:44.121145 (+ 23073us) tablet_metadata.cc:641] Metadata flushed
0626 19:58:44.124714 (+ 3569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":974666}
I20250626 19:58:44.132992 6854 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet f9adef69fe7442a4836a84934e792c05 (table default.table_a43e8ff4d1844c0b9dddf28fc310fd8f [id=ef672ba275b54933ae739bf9e560ae68]) successfully deleted
I20250626 19:58:44.134006 8102 tablet_replica.cc:331] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a: stopping tablet replica
I20250626 19:58:44.134876 8102 raft_consensus.cc:2241] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 19:58:44.135509 8102 raft_consensus.cc:2270] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 19:58:44.137501 8102 ts_tablet_manager.cc:1905] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 19:58:44.185755 8102 ts_tablet_manager.cc:1918] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 19:58:44.189579 8102 log.cc:1199] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a: Deleting WAL directory at /tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/ts-1/wal/wals/7b9a0f71535a4a65a775f728edbcc636
I20250626 19:58:44.194690 8102 ts_tablet_manager.cc:1939] T 7b9a0f71535a4a65a775f728edbcc636 P 315d304854c541bf8b5e6b1eea712d4a: Deleting consensus metadata
I20250626 19:58:44.200377 8102 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52438 (request call id 7) took 1101 ms. Trace:
I20250626 19:58:44.200829 8102 rpcz_store.cc:276] 0626 19:58:43.099197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:43.101518 (+ 2321us) service_pool.cc:225] Handling call
0626 19:58:44.133836 (+1032318us) ts_tablet_manager.cc:1145] Deleting tablet 7b9a0f71535a4a65a775f728edbcc636
0626 19:58:44.133889 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0626 19:58:44.145469 (+ 11580us) tablet_metadata.cc:641] Metadata flushed
0626 19:58:44.185525 (+ 40056us) tablet_metadata.cc:641] Metadata flushed
0626 19:58:44.200317 (+ 14792us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1031828}
I20250626 19:58:44.202250 6854 catalog_manager.cc:4928] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): tablet 7b9a0f71535a4a65a775f728edbcc636 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]) successfully deleted
I20250626 19:58:44.241648 6709 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=dac9e0aba8c943fabd12ce48a2dbd287]
I20250626 19:58:44.241925 7194 tablet_service.cc:1468] Processing CreateTablet for tablet 0084d8b5ae1548af8c943f8ace9bceb2 (DEFAULT_TABLE table=default.table_a43e8ff4d1844c0b9dddf28fc310fd8f [id=ef672ba275b54933ae739bf9e560ae68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:44.242465 6709 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 116) took 3782 ms. Trace:
I20250626 19:58:44.242928 7194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0084d8b5ae1548af8c943f8ace9bceb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:44.242764 6709 rpcz_store.cc:276] 0626 19:58:40.459919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:40.460036 (+ 117us) service_pool.cc:225] Handling call
0626 19:58:42.085517 (+1625481us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:42.087697 (+ 2180us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:42.691629 (+603932us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:42.693358 (+ 1729us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:42.693376 (+ 18us) write_op.cc:454] Released schema lock
0626 19:58:42.693888 (+ 512us) catalog_manager.cc:2156] Rolling back HMS table creation
0626 19:58:44.242384 (+1548496us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:42.693176 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 188d02ec81e645229acdc5eed98c9f9f is not leader of this config: current role FOLLOWER)
0626 19:58:42.693212 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:42.693232 (+ 20us) write_op.cc:454] Released schema lock
0626 19:58:42.693250 (+ 18us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1488753,"Hive Metastore.run_cpu_time_us":1173,"Hive Metastore.run_wall_time_us":570874,"child_traces":[["op",{}]]}
I20250626 19:58:44.247725 7341 tablet_service.cc:1468] Processing CreateTablet for tablet 0084d8b5ae1548af8c943f8ace9bceb2 (DEFAULT_TABLE table=default.table_a43e8ff4d1844c0b9dddf28fc310fd8f [id=ef672ba275b54933ae739bf9e560ae68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:44.249121 7341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0084d8b5ae1548af8c943f8ace9bceb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:44.250990 7339 tablet_service.cc:1468] Processing CreateTablet for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (DEFAULT_TABLE table=default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:44.251946 7339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aaf2ab9ed7840898ca54cca6983eb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:44.255366 7050 tablet_service.cc:1468] Processing CreateTablet for tablet 0084d8b5ae1548af8c943f8ace9bceb2 (DEFAULT_TABLE table=default.table_a43e8ff4d1844c0b9dddf28fc310fd8f [id=ef672ba275b54933ae739bf9e560ae68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:44.256261 7050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0084d8b5ae1548af8c943f8ace9bceb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:44.259491 7053 tablet_service.cc:1468] Processing CreateTablet for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (DEFAULT_TABLE table=default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:44.260484 7053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aaf2ab9ed7840898ca54cca6983eb82. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 19:58:44.264397 7418 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.46.188:38919)
I20250626 19:58:44.274950 7193 tablet_service.cc:1468] Processing CreateTablet for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (DEFAULT_TABLE table=default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 19:58:44.275933 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aaf2ab9ed7840898ca54cca6983eb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 19:58:44.288211 6867 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56016:
name: "default.Table_afaa8d56134e47008a0eb5b4de5c35da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:44.289186 8151 tablet_bootstrap.cc:492] T 0084d8b5ae1548af8c943f8ace9bceb2 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:44.302248 8151 tablet_bootstrap.cc:654] T 0084d8b5ae1548af8c943f8ace9bceb2 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:44.321749 8151 tablet_bootstrap.cc:492] T 0084d8b5ae1548af8c943f8ace9bceb2 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:44.322209 8151 ts_tablet_manager.cc:1397] T 0084d8b5ae1548af8c943f8ace9bceb2 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.004s
I20250626 19:58:44.332065 8151 raft_consensus.cc:357] T 0084d8b5ae1548af8c943f8ace9bceb2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:44.332718 8151 raft_consensus.cc:383] T 0084d8b5ae1548af8c943f8ace9bceb2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:44.332990 8151 raft_consensus.cc:738] T 0084d8b5ae1548af8c943f8ace9bceb2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:44.333678 8151 consensus_queue.cc:260] T 0084d8b5ae1548af8c943f8ace9bceb2 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
W20250626 19:58:44.383455 6869 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
I20250626 19:58:44.388924 6864 catalog_manager.cc:3460] renamed table default.table_63a5ae538eef4f739a22697dfcfb56c7 [id=22351693a59e46a9b9491eb79f1d0fe0] in HMS: new name default.table_be3ed77e691a4f68b7ddb861a1809419
I20250626 19:58:44.391269 8153 tablet_bootstrap.cc:492] T 0aaf2ab9ed7840898ca54cca6983eb82 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:44.392343 8151 ts_tablet_manager.cc:1428] T 0084d8b5ae1548af8c943f8ace9bceb2 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.066s user 0.006s sys 0.000s
F20250626 19:58:44.414661 6926 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 36, previous 36; dumping first 5 out of 8 received events: NotificationEvent(eventId=33, eventTime=1750967923, eventType=ALTER_TABLE, dbName=default, tableName=table_fd11e459e1d9471fae895dcdf01e89fa, message={"server":"","servicePrincipal":"","db":"default","table":"table_dc0e9650e86e4c60aec4f82f9b82f959","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_dc0e9650e86e4c60aec4f82f9b82f959\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750967919},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/warehouse/table_dc0e9650e86e4c60aec4f82f9b82f959\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_dc0e9650e86e4c60aec4f82f9b82f959\",\"kudu.table_id\":\"d1d807f596fd4519b4d74850a971205a\",\"transient_lastDdlTime\":\"1750967919\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"5cb1ee99f0744bcfa6ec1bff323536be\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_fd11e459e1d9471fae895dcdf01e89fa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750967919},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/warehouse/table_fd11e459e1d9471fae895dcdf01e89fa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fd11e459e1d9471fae895dcdf01e89fa\",\"kudu.table_id\":\"d1d807f596fd4519b4d74850a971205a\",\"transient_lastDdlTime\":\"1750967919\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"5cb1ee99f0744bcfa6ec1bff323536be\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750967923}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=34, eventTime=1750967923, eventType=DROP_TABLE, dbName=default, tableName=table_d04e336d056d484ab051754a2b012d9b, message={"server":"","servicePrincipal":"","db":"default","table":"table_d04e336d056d484ab051754a2b012d9b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d04e336d056d484ab051754a2b012d9b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750967914},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/warehouse/table_d04e336d056d484ab051754a2b012d9b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d04e336d056d484ab051754a2b012d9b\",\"kudu.table_id\":\"ad857025d66f4893872c8b5c92a9c0c7\",\"transient_lastDdlTime\":\"1750967914\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"5cb1ee99f0744bcfa6ec1bff323536be\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750967923}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=35, eventTime=1750967923, eventType=CREATE_TABLE, dbName=default, tableName=table_09c2ccbfee7e48e4a354a5138f0855d8, message={"server":"","servicePrincipal":"","db":"default","table":"table_09c2ccbfee7e48e4a354a5138f0855d8","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_09c2ccbfee7e48e4a354a5138f0855d8\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750967923},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/warehouse/table_09c2ccbfee7e48e4a354a5138f0855d8\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_09c2ccbfee7e48e4a354a5138f0855d8\",\"kudu.table_id\":\"bfe0cd9ec0ce4b009fbe0f0f7471c818\",\"transient_lastDdlTime\":\"1750967923\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"5cb1ee99f0744bcfa6ec1bff323536be\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750967923,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=36, eventTime=1750967923, eventType=CREATE_TABLE, dbName=default, tableName=table_df2c3f4a394c42cc81d626c1e46157f2, message={"server":"","servicePrincipal":"","db":"default","table":"table_df2c3f4a394c42cc81d626c1e46157f2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_df2c3f4a394c42cc81d626c1e46157f2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750967923},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/warehouse/table_df2c3f4a394c42cc81d626c1e46157f2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_df2c3f4a394c42cc81d626c1e46157f2\",\"kudu.table_id\":\"1e9ad2d55b0548498fb64172b882d9d0\",\"transient_lastDdlTime\":\"1750967923\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"5cb1ee99f0744bcfa6ec1bff323536be\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750967923,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=36, eventTime=1750967923, eventType=DROP_TABLE, dbName=default, tableName=table_a050ca8b9a9043ee87ca530228790f54, message={"server":"","servicePrincipal":"","db":"default","table":"table_a050ca8b9a9043ee87ca530228790f54","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a050ca8b9a9043ee87ca530228790f54\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750967913},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/warehouse/table_a050ca8b9a9043ee87ca530228790f54\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a050ca8b9a9043ee87ca530228790f54\",\"kudu.table_id\":\"def6cb7e52c748fcbe6c2ef2488d5f21\",\"transient_lastDdlTime\":\"1750967913\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"5cb1ee99f0744bcfa6ec1bff323536be\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750967923}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250626 19:58:44.422663 8151 tablet_bootstrap.cc:492] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a: Bootstrap starting.
I20250626 19:58:44.446940 8151 tablet_bootstrap.cc:654] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:44.447363 8155 tablet_bootstrap.cc:492] T 0084d8b5ae1548af8c943f8ace9bceb2 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:44.461864 8153 tablet_bootstrap.cc:654] T 0aaf2ab9ed7840898ca54cca6983eb82 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:44.479132 8155 tablet_bootstrap.cc:654] T 0084d8b5ae1548af8c943f8ace9bceb2 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:44.479946 8153 tablet_bootstrap.cc:492] T 0aaf2ab9ed7840898ca54cca6983eb82 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
@ 0x7fcafe46f950 google::LogMessage::Flush() at ??:0
I20250626 19:58:44.480377 8153 ts_tablet_manager.cc:1397] T 0aaf2ab9ed7840898ca54cca6983eb82 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.004s
I20250626 19:58:44.482838 8153 raft_consensus.cc:357] T 0aaf2ab9ed7840898ca54cca6983eb82 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:44.483547 8153 raft_consensus.cc:383] T 0aaf2ab9ed7840898ca54cca6983eb82 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:44.483840 8153 raft_consensus.cc:738] T 0aaf2ab9ed7840898ca54cca6983eb82 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:44.488754 6715 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=bfe0cd9ec0ce4b009fbe0f0f7471c818]
I20250626 19:58:44.489606 6715 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 115) took 4183 ms. Trace:
I20250626 19:58:44.489897 6715 rpcz_store.cc:276] 0626 19:58:40.306486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:40.306621 (+ 135us) service_pool.cc:225] Handling call
0626 19:58:42.091977 (+1785356us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:42.093977 (+ 2000us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:43.762870 (+1668893us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:43.765120 (+ 2250us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:43.765148 (+ 28us) write_op.cc:454] Released schema lock
0626 19:58:43.765683 (+ 535us) catalog_manager.cc:2156] Rolling back HMS table creation
0626 19:58:44.489542 (+723859us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:43.764511 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 188d02ec81e645229acdc5eed98c9f9f is not leader of this config: current role FOLLOWER)
0626 19:58:43.764675 (+ 164us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:43.764700 (+ 25us) write_op.cc:454] Released schema lock
0626 19:58:43.764722 (+ 22us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1641878,"Hive Metastore.run_cpu_time_us":986,"Hive Metastore.run_wall_time_us":748588,"child_traces":[["op",{}]]}
I20250626 19:58:44.491683 8151 tablet_bootstrap.cc:492] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a: No bootstrap required, opened a new log
I20250626 19:58:44.492134 8151 ts_tablet_manager.cc:1397] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.009s
I20250626 19:58:44.496881 8151 raft_consensus.cc:357] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:44.497617 8151 raft_consensus.cc:383] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:44.498644 6869 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56016:
name: "default.Table_09c2ccbfee7e48e4a354a5138f0855d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:44.497888 8151 raft_consensus.cc:738] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d304854c541bf8b5e6b1eea712d4a, State: Initialized, Role: FOLLOWER
I20250626 19:58:44.499696 8153 consensus_queue.cc:260] T 0aaf2ab9ed7840898ca54cca6983eb82 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:44.503363 8153 ts_tablet_manager.cc:1428] T 0aaf2ab9ed7840898ca54cca6983eb82 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.023s user 0.005s sys 0.002s
I20250626 19:58:44.504238 8153 tablet_bootstrap.cc:492] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca: Bootstrap starting.
I20250626 19:58:44.508880 6863 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
W20250626 19:58:44.510663 6863 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 0aaf2ab9ed7840898ca54cca6983eb82 consensus state has no leader
I20250626 19:58:44.515959 8151 consensus_queue.cc:260] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:44.536782 6863 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
I20250626 19:58:44.538355 8155 tablet_bootstrap.cc:492] T 0084d8b5ae1548af8c943f8ace9bceb2 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:44.538839 8155 ts_tablet_manager.cc:1397] T 0084d8b5ae1548af8c943f8ace9bceb2 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.092s user 0.008s sys 0.004s
I20250626 19:58:44.540364 8151 ts_tablet_manager.cc:1428] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a: Time spent starting tablet: real 0.048s user 0.006s sys 0.001s
I20250626 19:58:44.548945 6868 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
@ 0x7fcafe46f6bf google::LogMessage::~LogMessage() at ??:0
I20250626 19:58:44.543493 8155 raft_consensus.cc:357] T 0084d8b5ae1548af8c943f8ace9bceb2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:44.565891 8155 raft_consensus.cc:383] T 0084d8b5ae1548af8c943f8ace9bceb2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:44.566236 8155 raft_consensus.cc:738] T 0084d8b5ae1548af8c943f8ace9bceb2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:44.567049 8155 consensus_queue.cc:260] T 0084d8b5ae1548af8c943f8ace9bceb2 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:44.569496 8155 ts_tablet_manager.cc:1428] T 0084d8b5ae1548af8c943f8ace9bceb2 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20250626 19:58:44.577507 8155 tablet_bootstrap.cc:492] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7: Bootstrap starting.
I20250626 19:58:44.618950 6863 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
I20250626 19:58:44.619326 8155 tablet_bootstrap.cc:654] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:44.628240 8153 tablet_bootstrap.cc:654] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca: Neither blocks nor log segments found. Creating new log.
I20250626 19:58:44.643775 8155 tablet_bootstrap.cc:492] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7: No bootstrap required, opened a new log
I20250626 19:58:44.644215 8155 ts_tablet_manager.cc:1397] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7: Time spent bootstrapping tablet: real 0.067s user 0.005s sys 0.005s
I20250626 19:58:44.646622 8155 raft_consensus.cc:357] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:44.647300 8155 raft_consensus.cc:383] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:44.647553 8155 raft_consensus.cc:738] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297d2c913667408bae82668a557e06c7, State: Initialized, Role: FOLLOWER
I20250626 19:58:44.660854 8155 consensus_queue.cc:260] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:44.665542 8153 tablet_bootstrap.cc:492] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca: No bootstrap required, opened a new log
I20250626 19:58:44.665964 8153 ts_tablet_manager.cc:1397] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca: Time spent bootstrapping tablet: real 0.162s user 0.004s sys 0.008s
I20250626 19:58:44.668332 8153 raft_consensus.cc:357] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:44.668956 8153 raft_consensus.cc:383] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 19:58:44.669221 8153 raft_consensus.cc:738] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Initialized, Role: FOLLOWER
I20250626 19:58:44.673735 8155 ts_tablet_manager.cc:1428] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20250626 19:58:44.674237 6863 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
I20250626 19:58:44.688730 6863 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
I20250626 19:58:44.669874 8153 consensus_queue.cc:260] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
@ 0x7fcb0934a63c kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250626 19:58:44.724097 8153 ts_tablet_manager.cc:1428] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca: Time spent starting tablet: real 0.058s user 0.005s sys 0.000s
I20250626 19:58:44.733213 6711 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=1e9ad2d55b0548498fb64172b882d9d0]
I20250626 19:58:44.734050 6711 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 117) took 4120 ms. Trace:
I20250626 19:58:44.734349 6711 rpcz_store.cc:276] 0626 19:58:40.613514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:40.613660 (+ 146us) service_pool.cc:225] Handling call
0626 19:58:42.171738 (+1558078us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:42.173789 (+ 2051us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:43.986341 (+1812552us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:43.988238 (+ 1897us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:43.988285 (+ 47us) write_op.cc:454] Released schema lock
0626 19:58:43.988800 (+ 515us) catalog_manager.cc:2156] Rolling back HMS table creation
0626 19:58:44.733990 (+745190us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:43.988049 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 188d02ec81e645229acdc5eed98c9f9f is not leader of this config: current role FOLLOWER)
0626 19:58:43.988085 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:43.988108 (+ 23us) write_op.cc:454] Released schema lock
0626 19:58:43.988128 (+ 20us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":2087787,"Hive Metastore.run_cpu_time_us":942,"Hive Metastore.run_wall_time_us":467336,"mutex_wait_us":74053,"child_traces":[["op",{}]]}
I20250626 19:58:44.748425 6868 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56016:
name: "default.Table_df2c3f4a394c42cc81d626c1e46157f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fcb09349dfc kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250626 19:58:44.784907 8137 consensus_queue.cc:1035] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:44.798005 6716 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=a187bf25ba884923869c7367c1d00dc4]
I20250626 19:58:44.799008 6716 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42822 (request call id 119) took 4046 ms. Trace:
I20250626 19:58:44.799314 6716 rpcz_store.cc:276] 0626 19:58:40.752408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 19:58:40.772559 (+ 20151us) service_pool.cc:225] Handling call
0626 19:58:42.175858 (+1403299us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 19:58:42.177556 (+ 1698us) catalog_manager.cc:2122] Created new table and tablet info
0626 19:58:44.115031 (+1937475us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 19:58:44.116750 (+ 1719us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:44.116770 (+ 20us) write_op.cc:454] Released schema lock
0626 19:58:44.117281 (+ 511us) catalog_manager.cc:2156] Rolling back HMS table creation
0626 19:58:44.798943 (+681662us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 19:58:44.116561 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 188d02ec81e645229acdc5eed98c9f9f is not leader of this config: current role FOLLOWER)
0626 19:58:44.116595 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0626 19:58:44.116615 (+ 20us) write_op.cc:454] Released schema lock
0626 19:58:44.116634 (+ 19us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":2404471,"Hive Metastore.run_cpu_time_us":1001,"Hive Metastore.run_wall_time_us":193658,"mutex_wait_us":51737,"child_traces":[["op",{}]]}
I20250626 19:58:44.816371 6870 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56016:
name: "default.Table_1d890136b93b48cab4d218cbd6a72bba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fcb0934de72 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250626 19:58:44.819985 8166 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6610553ba0df459085f7373935774377)
I20250626 19:58:44.820406 8166 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:44.822453 8166 leader_election.cc:290] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
I20250626 19:58:44.822829 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 215 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f" is_pre_election: true
I20250626 19:58:44.823534 6728 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae38ec946e14198ae4adc8dde9d66e8 in term 2.
I20250626 19:58:44.824672 7756 leader_election.cc:304] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 188d02ec81e645229acdc5eed98c9f9f, 8ae38ec946e14198ae4adc8dde9d66e8; no voters:
I20250626 19:58:44.825233 6880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 215 } } ignore_live_leader: false dest_uuid: "6610553ba0df459085f7373935774377" is_pre_election: true
I20250626 19:58:44.825414 8166 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 19:58:44.825680 8166 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6610553ba0df459085f7373935774377)
I20250626 19:58:44.825958 8166 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 2 FOLLOWER]: Advancing to term 3
I20250626 19:58:44.833580 8166 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:44.836057 6728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 215 } } ignore_live_leader: false dest_uuid: "188d02ec81e645229acdc5eed98c9f9f"
I20250626 19:58:44.836632 6728 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 2 FOLLOWER]: Advancing to term 3
W20250626 19:58:44.843704 6862 catalog_manager.cc:5415] Ignoring report from unknown tablet 089e49e29b2b4f3dbea8a4597d9e36a2
I20250626 19:58:44.848793 6728 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae38ec946e14198ae4adc8dde9d66e8 in term 3.
I20250626 19:58:44.850644 7756 leader_election.cc:304] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 188d02ec81e645229acdc5eed98c9f9f, 8ae38ec946e14198ae4adc8dde9d66e8; no voters:
I20250626 19:58:44.857419 6880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 215 } } ignore_live_leader: false dest_uuid: "6610553ba0df459085f7373935774377"
I20250626 19:58:44.860335 8166 leader_election.cc:290] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [CANDIDATE]: Term 3 election: Requested vote from peers 188d02ec81e645229acdc5eed98c9f9f (127.4.46.190:39311), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
I20250626 19:58:44.860874 8166 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 19:58:44.861971 8166 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 3 LEADER]: Becoming Leader. State: Replica: 8ae38ec946e14198ae4adc8dde9d66e8, State: Running, Role: LEADER
I20250626 19:58:44.862864 8166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 2.215, Last appended by leader: 215, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:44.859786 6728 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Rejecting Update request from peer 6610553ba0df459085f7373935774377 for earlier term 2. Current term is 3. Ops: []
I20250626 19:58:44.865933 8167 raft_consensus.cc:491] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
@ 0x7fcb0934de2a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20250626 19:58:44.866375 8167 raft_consensus.cc:513] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:44.876726 6880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "188d02ec81e645229acdc5eed98c9f9f" candidate_term: 4 candidate_status { last_received { term: 2 index: 215 } } ignore_live_leader: false dest_uuid: "6610553ba0df459085f7373935774377" is_pre_election: true
I20250626 19:58:44.872144 8171 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 }, Status: INVALID_TERM, Last received: 2.215, Next index: 216, Last known committed idx: 215, Time since last communication: 0.006s
I20250626 19:58:44.881798 8138 consensus_queue.cc:1035] T cbac3cf560fb4f41ac7ddac25bcb9e5b P 315d304854c541bf8b5e6b1eea712d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:44.883963 8166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ae38ec946e14198ae4adc8dde9d66e8. Latest consensus state: current_term: 3 leader_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:44.884606 8166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: This master's current role is: LEADER
I20250626 19:58:44.886818 7814 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "188d02ec81e645229acdc5eed98c9f9f" candidate_term: 4 candidate_status { last_received { term: 2 index: 215 } } ignore_live_leader: false dest_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" is_pre_election: true
I20250626 19:58:44.886691 8167 leader_election.cc:290] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
I20250626 19:58:44.891978 6705 leader_election.cc:304] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 188d02ec81e645229acdc5eed98c9f9f; no voters: 6610553ba0df459085f7373935774377, 8ae38ec946e14198ae4adc8dde9d66e8
@ 0x7fcb0934ddba std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20250626 19:58:44.902530 8167 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 19:58:44.909039 8086 raft_consensus.cc:491] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:44.909556 8086 raft_consensus.cc:513] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:44.912429 8086 leader_election.cc:290] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:44.915513 7066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0aaf2ab9ed7840898ca54cca6983eb82" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:44.916107 7066 raft_consensus.cc:2466] T 0aaf2ab9ed7840898ca54cca6983eb82 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
I20250626 19:58:44.918547 7294 leader_election.cc:304] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:44.921514 8086 raft_consensus.cc:2802] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:44.921931 8086 raft_consensus.cc:491] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:44.922533 8086 raft_consensus.cc:3058] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:44.941431 8086 raft_consensus.cc:513] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:44.949155 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0aaf2ab9ed7840898ca54cca6983eb82" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:44.949692 7215 raft_consensus.cc:2466] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 0.
@ 0x7fcb0934dd82 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20250626 19:58:44.955199 7066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0aaf2ab9ed7840898ca54cca6983eb82" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca"
I20250626 19:58:44.955725 7066 raft_consensus.cc:3058] T 0aaf2ab9ed7840898ca54cca6983eb82 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:44.956933 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0aaf2ab9ed7840898ca54cca6983eb82" candidate_uuid: "297d2c913667408bae82668a557e06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
I20250626 19:58:44.957393 7215 raft_consensus.cc:3058] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:44.965468 7215 raft_consensus.cc:2466] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:44.966696 7293 leader_election.cc:304] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a; no voters:
I20250626 19:58:44.966943 8137 raft_consensus.cc:491] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:44.967296 8137 raft_consensus.cc:513] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:44.967797 8086 leader_election.cc:290] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:44.968354 8086 raft_consensus.cc:2802] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:44.968767 8086 raft_consensus.cc:695] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 297d2c913667408bae82668a557e06c7, State: Running, Role: LEADER
I20250626 19:58:44.969617 7073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0aaf2ab9ed7840898ca54cca6983eb82" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe151be01e24c82854d9c8c79c797ca" is_pre_election: true
I20250626 19:58:44.969589 8086 consensus_queue.cc:237] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:44.973230 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0aaf2ab9ed7840898ca54cca6983eb82" candidate_uuid: "315d304854c541bf8b5e6b1eea712d4a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:44.976121 7152 leader_election.cc:304] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a; no voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca
I20250626 19:58:44.976938 8138 raft_consensus.cc:2747] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 19:58:44.962637 7066 raft_consensus.cc:2466] T 0aaf2ab9ed7840898ca54cca6983eb82 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297d2c913667408bae82668a557e06c7 in term 1.
I20250626 19:58:44.982666 8137 leader_election.cc:290] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 19:58:44.989207 8174 log.cc:927] Time spent T cbac3cf560fb4f41ac7ddac25bcb9e5b P 297d2c913667408bae82668a557e06c7: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20250626 19:58:45.007817 8154 raft_consensus.cc:491] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:45.008268 8154 raft_consensus.cc:513] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:45.010147 8154 leader_election.cc:290] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:45.011927 7359 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0084d8b5ae1548af8c943f8ace9bceb2" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7" is_pre_election: true
I20250626 19:58:45.012055 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0084d8b5ae1548af8c943f8ace9bceb2" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a" is_pre_election: true
I20250626 19:58:45.012495 7359 raft_consensus.cc:2466] T 0084d8b5ae1548af8c943f8ace9bceb2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:45.012599 7215 raft_consensus.cc:2466] T 0084d8b5ae1548af8c943f8ace9bceb2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 0.
I20250626 19:58:45.013443 7006 leader_election.cc:304] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315d304854c541bf8b5e6b1eea712d4a, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:45.014592 8154 raft_consensus.cc:2802] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 19:58:45.014950 8154 raft_consensus.cc:491] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 19:58:45.015286 8154 raft_consensus.cc:3058] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:45.016098 7814 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [term 3 LEADER]: Rejecting Update request from peer 6610553ba0df459085f7373935774377 for earlier term 2. Current term is 3. Ops: []
I20250626 19:58:45.016597 8177 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 19:58:45.016911 8177 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6610553ba0df459085f7373935774377, State: Running, Role: LEADER
I20250626 19:58:45.022168 8154 raft_consensus.cc:513] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:45.023212 8175 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 19:58:45.024649 7360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0084d8b5ae1548af8c943f8ace9bceb2" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297d2c913667408bae82668a557e06c7"
I20250626 19:58:45.024559 8177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 2.215, Last appended by leader: 215, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:45.025153 7360 raft_consensus.cc:3058] T 0084d8b5ae1548af8c943f8ace9bceb2 P 297d2c913667408bae82668a557e06c7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:45.025677 8177 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [term 2 FOLLOWER]: Advancing to term 3
W20250626 19:58:45.031308 8173 consensus_queue.cc:1162] T 00000000000000000000000000000000 P 6610553ba0df459085f7373935774377 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" responder_term: 3 status { last_received { term: 3 index: 216 } last_committed_idx: 215 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6610553ba0df459085f7373935774377 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250626 19:58:45.033849 7360 raft_consensus.cc:2466] T 0084d8b5ae1548af8c943f8ace9bceb2 P 297d2c913667408bae82668a557e06c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
I20250626 19:58:45.034982 7009 leader_election.cc:304] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297d2c913667408bae82668a557e06c7, bbe151be01e24c82854d9c8c79c797ca; no voters:
I20250626 19:58:45.035305 6863 catalog_manager.cc:5582] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:45.036625 6863 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
@ 0x7fcb0934d07e std::__1::__function::__func<>::operator()() at ??:0
@ 0x59bcd5 std::__1::__function::__value_func<>::operator()() at ??:?
E20250626 19:58:45.060058 6863 catalog_manager.cc:5737] Error updating tablets from 297d2c913667408bae82668a557e06c7: Illegal state: replica 6610553ba0df459085f7373935774377 is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "0aaf2ab9ed7840898ca54cca6983eb82" state: RUNNING consensus_state { current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } sequence_number: 191
I20250626 19:58:45.061954 8154 leader_election.cc:290] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 election: Requested vote from peers 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703), 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373)
I20250626 19:58:45.061892 7215 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0084d8b5ae1548af8c943f8ace9bceb2" candidate_uuid: "bbe151be01e24c82854d9c8c79c797ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d304854c541bf8b5e6b1eea712d4a"
W20250626 19:58:45.062228 6856 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 9d677beae63a49ecbd11ed53e9e421fc (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9d677beae63a49ecbd11ed53e9e421fc consensus state has no leader
I20250626 19:58:45.062410 7215 raft_consensus.cc:3058] T 0084d8b5ae1548af8c943f8ace9bceb2 P 315d304854c541bf8b5e6b1eea712d4a [term 0 FOLLOWER]: Advancing to term 1
I20250626 19:58:45.062463 8154 raft_consensus.cc:2802] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [term 1 FOLLOWER]: Leader election won for term 1
I20250626 19:58:45.062896 8154 raft_consensus.cc:695] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [term 1 LEADER]: Becoming Leader. State: Replica: bbe151be01e24c82854d9c8c79c797ca, State: Running, Role: LEADER
@ 0x59bb0a std::__1::function<>::operator()() at ??:?
I20250626 19:58:45.066977 8154 consensus_queue.cc:237] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } }
I20250626 19:58:45.068921 7215 raft_consensus.cc:2466] T 0084d8b5ae1548af8c943f8ace9bceb2 P 315d304854c541bf8b5e6b1eea712d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe151be01e24c82854d9c8c79c797ca in term 1.
@ 0x7fcb0068a637 kudu::Thread::SuperviseThread() at ??:0
@ 0x465e60 __tsan_thread_start_func at ??:?
I20250626 19:58:45.111687 8175 catalog_manager.cc:671] Loaded metadata for table default.table_acc449d45fc742dab0b039b077cc75e0 [id=11ee0dd5a39742399865b36ea597920d]
I20250626 19:58:45.119463 8175 catalog_manager.cc:671] Loaded metadata for table default.table_63a5ae538eef4f739a22697dfcfb56c7 [id=22351693a59e46a9b9491eb79f1d0fe0]
I20250626 19:58:45.120997 8175 catalog_manager.cc:671] Loaded metadata for table default.table_ced1a26248c748fb8c06cd348d12fb19 [id=3f20c2a68db64ff7afbc98716ecf20b0]
I20250626 19:58:45.122483 8175 catalog_manager.cc:671] Loaded metadata for table default.table_6bf87f28d4d34102bf1d02e2e1465a33 [id=58d985607f30440ea82dab74e20a1f8f]
I20250626 19:58:45.129492 8175 catalog_manager.cc:671] Loaded metadata for table default.table_30a8077ecd80407991c244d577d16cad [id=7c0b8a662d1442b5b758ace1e136d79a]
I20250626 19:58:45.131037 8175 catalog_manager.cc:671] Loaded metadata for table default.table_fa425acbc3844433a95ac1c8ee1dc26d [id=853420b7ab29422e9683dd9e353f87af]
I20250626 19:58:45.132527 8175 catalog_manager.cc:671] Loaded metadata for table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]
I20250626 19:58:45.143159 8175 catalog_manager.cc:671] Loaded metadata for table default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d]
I20250626 19:58:45.145561 8175 catalog_manager.cc:671] Loaded metadata for table default.table_6f04f2a6bf414cb8912fe28aff625d2e [id=912c0fafb9264e07b1fa485ea0d193d9]
I20250626 19:58:45.147158 8175 catalog_manager.cc:671] Loaded metadata for table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7]
I20250626 19:58:45.155990 8175 catalog_manager.cc:671] Loaded metadata for table default.table_dc0e9650e86e4c60aec4f82f9b82f959 [id=d1d807f596fd4519b4d74850a971205a]
I20250626 19:58:45.159567 8175 catalog_manager.cc:671] Loaded metadata for table default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21]
I20250626 19:58:45.161150 8175 catalog_manager.cc:671] Loaded metadata for table default.table_a43e8ff4d1844c0b9dddf28fc310fd8f [id=ef672ba275b54933ae739bf9e560ae68]
@ 0x7fcb01d096db start_thread at ??:0
@ 0x7fcafd11471f clone at ??:0
I20250626 19:58:45.264348 6728 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Refusing update from remote peer 8ae38ec946e14198ae4adc8dde9d66e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 215. Preceding OpId from leader: term: 3 index: 216. (index mismatch)
I20250626 19:58:45.265759 8166 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20250626 19:58:45.269305 8167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ae38ec946e14198ae4adc8dde9d66e8. Latest consensus state: current_term: 3 leader_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:45.269948 8167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:58:45.284812 8175 tablet_loader.cc:96] loaded metadata for tablet 0084d8b5ae1548af8c943f8ace9bceb2 (table default.table_a43e8ff4d1844c0b9dddf28fc310fd8f [id=ef672ba275b54933ae739bf9e560ae68])
I20250626 19:58:45.293859 8172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:45.294617 8172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 [sys.catalog]: This master's current role is: LEADER
I20250626 19:58:45.295886 8175 tablet_loader.cc:96] loaded metadata for tablet 0487e1d34dd74d70a5b5c8ac8cc0f636 (table default.table_6f04f2a6bf414cb8912fe28aff625d2e [id=912c0fafb9264e07b1fa485ea0d193d9])
I20250626 19:58:45.296296 8167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } } }
I20250626 19:58:45.296906 8167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [sys.catalog]: This master's current role is: FOLLOWER
I20250626 19:58:45.297268 8175 tablet_loader.cc:96] loaded metadata for tablet 08c4b36ad87c47f2a72a3118c3a66fd1 (table default.table_d04e336d056d484ab051754a2b012d9b [id=ad857025d66f4893872c8b5c92a9c0c7])
I20250626 19:58:45.304140 8175 tablet_loader.cc:96] loaded metadata for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49])
I20250626 19:58:45.305459 8175 tablet_loader.cc:96] loaded metadata for tablet 196180f1e78a4d5ca8aca8d7bb666d42 (table default.table_dc0e9650e86e4c60aec4f82f9b82f959 [id=d1d807f596fd4519b4d74850a971205a])
I20250626 19:58:45.306744 8175 tablet_loader.cc:96] loaded metadata for tablet 2ced063ac30b4168b4af4cf2ba239e25 (table default.table_acc449d45fc742dab0b039b077cc75e0 [id=11ee0dd5a39742399865b36ea597920d])
I20250626 19:58:45.308058 8175 tablet_loader.cc:96] loaded metadata for tablet 40b6570e38434e0c8301d010e96c67a3 (table default.table_ced1a26248c748fb8c06cd348d12fb19 [id=3f20c2a68db64ff7afbc98716ecf20b0])
I20250626 19:58:45.324414 8175 tablet_loader.cc:96] loaded metadata for tablet 55fad5c8639047a0a5519243d6ae76a3 (table default.table_6bf87f28d4d34102bf1d02e2e1465a33 [id=58d985607f30440ea82dab74e20a1f8f])
I20250626 19:58:45.325917 8175 tablet_loader.cc:96] loaded metadata for tablet 7bf0c724d4684773b40aff24524e1e92 (table default.table_64bb792c0fa74fab8347bf7bda0dedbc [id=8c650357a157442e8b725ae57770ac8d])
I20250626 19:58:45.327389 8175 tablet_loader.cc:96] loaded metadata for tablet 9de12a70dc90406c9ca8f58d253f35d5 (table default.table_fa425acbc3844433a95ac1c8ee1dc26d [id=853420b7ab29422e9683dd9e353f87af])
I20250626 19:58:45.328814 8175 tablet_loader.cc:96] loaded metadata for tablet c7159554b45442b5aa5069d7a47d1f8e (table default.table_30a8077ecd80407991c244d577d16cad [id=7c0b8a662d1442b5b758ace1e136d79a])
I20250626 19:58:45.330265 8175 tablet_loader.cc:96] loaded metadata for tablet cbac3cf560fb4f41ac7ddac25bcb9e5b (table default.table_63a5ae538eef4f739a22697dfcfb56c7 [id=22351693a59e46a9b9491eb79f1d0fe0])
I20250626 19:58:45.355922 8175 tablet_loader.cc:96] loaded metadata for tablet fabf0275e32e400c9731eb09d9c2620c (table default.table_a050ca8b9a9043ee87ca530228790f54 [id=def6cb7e52c748fcbe6c2ef2488d5f21])
I20250626 19:58:45.359256 8175 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 19:58:45.425459 8175 catalog_manager.cc:1261] Loaded cluster ID: 5cb1ee99f0744bcfa6ec1bff323536be
I20250626 19:58:45.425797 8175 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
W20250626 19:58:45.467566 6705 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 17d9cd26171943088ad676722edf9577 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 17d9cd26171943088ad676722edf9577 consensus state has no leader
I20250626 19:58:45.472383 8175 catalog_manager.cc:1506] Loading token signing keys...
I20250626 19:58:45.487684 8175 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8: Loaded TSK: 0
I20250626 19:58:45.488929 8175 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 19:58:45.525964 8175 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250626 19:58:45.541966 8175 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 32
I20250626 19:58:45.735692 8086 consensus_queue.cc:1035] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
F20250626 19:58:45.754740 8020 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 36, previous 36; dumping first 5 out of 11 received events: NotificationEvent(eventId=33, eventTime=1750967923, eventType=ALTER_TABLE, dbName=default, tableName=table_fd11e459e1d9471fae895dcdf01e89fa, message={"server":"","servicePrincipal":"","db":"default","table":"table_dc0e9650e86e4c60aec4f82f9b82f959","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_dc0e9650e86e4c60aec4f82f9b82f959\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750967919},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/warehouse/table_dc0e9650e86e4c60aec4f82f9b82f959\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_dc0e9650e86e4c60aec4f82f9b82f959\",\"kudu.table_id\":\"d1d807f596fd4519b4d74850a971205a\",\"transient_lastDdlTime\":\"1750967919\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"5cb1ee99f0744bcfa6ec1bff323536be\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_fd11e459e1d9471fae895dcdf01e89fa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750967919},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/warehouse/table_fd11e459e1d9471fae895dcdf01e89fa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fd11e459e1d9471fae895dcdf01e89fa\",\"kudu.table_id\":\"d1d807f596fd4519b4d74850a971205a\",\"transient_lastDdlTime\":\"1750967919\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"5cb1ee99f0744bcfa6ec1bff323536be\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750967923}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=34, eventTime=1750967923, eventType=DROP_TABLE, dbName=default, tableName=table_d04e336d056d484ab051754a2b012d9b, message={"server":"","servicePrincipal":"","db":"default","table":"table_d04e336d056d484ab051754a2b012d9b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d04e336d056d484ab051754a2b012d9b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750967914},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/warehouse/table_d04e336d056d484ab051754a2b012d9b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d04e336d056d484ab051754a2b012d9b\",\"kudu.table_id\":\"ad857025d66f4893872c8b5c92a9c0c7\",\"transient_lastDdlTime\":\"1750967914\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"5cb1ee99f0744bcfa6ec1bff323536be\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750967923}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=35, eventTime=1750967923, eventType=CREATE_TABLE, dbName=default, tableName=table_09c2ccbfee7e48e4a354a5138f0855d8, message={"server":"","servicePrincipal":"","db":"default","table":"table_09c2ccbfee7e48e4a354a5138f0855d8","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_09c2ccbfee7e48e4a354a5138f0855d8\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750967923},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/warehouse/table_09c2ccbfee7e48e4a354a5138f0855d8\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_09c2ccbfee7e48e4a354a5138f0855d8\",\"kudu.table_id\":\"bfe0cd9ec0ce4b009fbe0f0f7471c818\",\"transient_lastDdlTime\":\"1750967923\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"5cb1ee99f0744bcfa6ec1bff323536be\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750967923,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=36, eventTime=1750967923, eventType=CREATE_TABLE, dbName=default, tableName=table_df2c3f4a394c42cc81d626c1e46157f2, message={"server":"","servicePrincipal":"","db":"default","table":"table_df2c3f4a394c42cc81d626c1e46157f2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_df2c3f4a394c42cc81d626c1e46157f2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750967923},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/warehouse/table_df2c3f4a394c42cc81d626c1e46157f2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_df2c3f4a394c42cc81d626c1e46157f2\",\"kudu.table_id\":\"1e9ad2d55b0548498fb64172b882d9d0\",\"transient_lastDdlTime\":\"1750967923\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"5cb1ee99f0744bcfa6ec1bff323536be\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750967923,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=36, eventTime=1750967923, eventType=DROP_TABLE, dbName=default, tableName=table_a050ca8b9a9043ee87ca530228790f54, message={"server":"","servicePrincipal":"","db":"default","table":"table_a050ca8b9a9043ee87ca530228790f54","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a050ca8b9a9043ee87ca530228790f54\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750967913},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskAJMAmr/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1750967830332128-4282-0/minicluster-data/warehouse/table_a050ca8b9a9043ee87ca530228790f54\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a050ca8b9a9043ee87ca530228790f54\",\"kudu.table_id\":\"def6cb7e52c748fcbe6c2ef2488d5f21\",\"transient_lastDdlTime\":\"1750967913\",\"comment\":\"\",\"kudu.master_addresses\":\"127.4.46.190:39311,127.4.46.189:36417,127.4.46.188:38919\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"5cb1ee99f0744bcfa6ec1bff323536be\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750967923}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250626 19:58:45.770516 8086 consensus_queue.cc:1035] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 19:58:45.775431 8167 raft_consensus.cc:491] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8ae38ec946e14198ae4adc8dde9d66e8)
I20250626 19:58:45.775900 8167 raft_consensus.cc:513] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:45.777740 8167 leader_election.cc:290] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
I20250626 19:58:45.778398 7814 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "188d02ec81e645229acdc5eed98c9f9f" candidate_term: 4 candidate_status { last_received { term: 3 index: 216 } } ignore_live_leader: false dest_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" is_pre_election: true
I20250626 19:58:45.777127 7785 catalog_manager.cc:5582] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 297d2c913667408bae82668a557e06c7 (127.4.46.131). New cstate: current_term: 1 leader_uuid: "297d2c913667408bae82668a557e06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } }
I20250626 19:58:45.789602 7785 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
I20250626 19:58:45.805353 7119 heartbeater.cc:499] Master 127.4.46.189:36417 was elected leader, sending a full tablet report...
W20250626 19:58:45.835949 7787 catalog_manager.cc:5415] Ignoring report from unknown tablet 089e49e29b2b4f3dbea8a4597d9e36a2
W20250626 19:58:45.836843 7787 catalog_manager.cc:5415] Ignoring report from unknown tablet ae04997b77924e8ba479c5c34ac822d7
W20250626 19:58:45.838019 7787 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
@ 0x7f65c0867950 google::LogMessage::Flush() at ??:0
@ 0x7f65c08676bf google::LogMessage::~LogMessage() at ??:0
@ 0x7f65cb74263c kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f65cb741dfc kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
W20250626 19:58:45.988196 7420 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.46.188:38919)
W20250626 19:58:45.989436 7420 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.46.188:38919)
I20250626 19:58:45.993546 7262 heartbeater.cc:499] Master 127.4.46.189:36417 was elected leader, sending a full tablet report...
I20250626 19:58:45.995822 7782 catalog_manager.cc:5674] TS 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703) does not have the latest schema for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
@ 0x7f65cb745e72 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250626 19:58:46.013303 7786 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39232:
table { table_name: "default.Table_63a5ae538eef4f739a22697dfcfb56c7" } new_table_name: "default.Table_be3ed77e691a4f68b7ddb861a1809419" modify_external_catalogs: true
I20250626 19:58:46.013381 7785 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39232:
name: "default.Table_09c2ccbfee7e48e4a354a5138f0855d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:46.017374 7782 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39232:
name: "default.Table_df2c3f4a394c42cc81d626c1e46157f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 19:58:46.028342 7420 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.4.46.189:36417)
I20250626 19:58:46.031076 7780 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39232:
name: "default.Table_afaa8d56134e47008a0eb5b4de5c35da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:46.032538 7787 catalog_manager.cc:5582] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca reported cstate change: term changed from 0 to 1, leader changed from <none> to bbe151be01e24c82854d9c8c79c797ca (127.4.46.129). New cstate: current_term: 1 leader_uuid: "bbe151be01e24c82854d9c8c79c797ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } health_report { overall_health: UNKNOWN } } }
I20250626 19:58:46.034895 7779 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39232:
name: "default.Table_1d890136b93b48cab4d218cbd6a72bba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 19:58:46.035125 7775 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39232:
table { table_name: "default.Table_a050ca8b9a9043ee87ca530228790f54" } modify_external_catalogs: true
I20250626 19:58:46.036860 7775 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39232:
table { table_name: "default.Table_a050ca8b9a9043ee87ca530228790f54" } modify_external_catalogs: true
I20250626 19:58:46.038621 7787 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
W20250626 19:58:46.041909 7771 catalog_manager.cc:5415] Ignoring report from unknown tablet 089e49e29b2b4f3dbea8a4597d9e36a2
W20250626 19:58:46.045979 7152 connection.cc:537] client connection to 127.4.46.188:38919 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 19:58:46.047890 7296 connection.cc:537] client connection to 127.4.46.188:38919 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250626 19:58:46.053202 7769 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39232:
table { table_name: "default.Table_d04e336d056d484ab051754a2b012d9b" } modify_external_catalogs: true
I20250626 19:58:46.055473 7769 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39232:
table { table_name: "default.Table_d04e336d056d484ab051754a2b012d9b" } modify_external_catalogs: true
W20250626 19:58:46.055943 7009 connection.cc:537] client connection to 127.4.46.188:38919 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 19:58:46.058116 7771 catalog_manager.cc:5415] Ignoring report from unknown tablet e267bd75d7d342a3a6c28b17eb37d83e
W20250626 19:58:46.058634 7771 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
W20250626 19:58:46.059301 7771 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
W20250626 19:58:46.066583 6705 connection.cc:537] client connection to 127.4.46.188:38919 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 19:58:46.067042 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 4 similar messages]
W20250626 19:58:46.073096 7759 connection.cc:537] client connection to 127.4.46.188:38919 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 19:58:46.073587 7759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x7f65cb745e2a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
W20250626 19:58:46.085546 8218 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 19:58:46.041726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.46.188:38919 (local address 127.0.0.1:60716)
0626 19:58:46.073687 (+ 31961us) negotiation.cc:107] Waiting for socket to connect
0626 19:58:46.073741 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0626 19:58:46.074066 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0626 19:58:46.084409 (+ 10343us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":31212,"thread_start_us":40069,"threads_started":1}
W20250626 19:58:46.085841 8220 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 19:58:46.050560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.46.188:38919 (local address 127.4.46.131:59495)
0626 19:58:46.062599 (+ 12039us) negotiation.cc:107] Waiting for socket to connect
0626 19:58:46.062667 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0626 19:58:46.063097 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0626 19:58:46.084384 (+ 21287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":11126,"thread_start_us":15077,"threads_started":1}
I20250626 19:58:46.095793 7771 catalog_manager.cc:5674] TS 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373) does not have the latest schema for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
@ 0x7f65cb745dba std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
W20250626 19:58:46.127800 7759 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.188:38919, user_credentials={real_user=slave}} blocked reactor thread for 54267.5us
W20250626 19:58:46.131034 8219 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 19:58:46.047925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.46.188:38919 (local address 127.4.46.130:43265)
0626 19:58:46.130120 (+ 82195us) negotiation.cc:107] Waiting for socket to connect
0626 19:58:46.130340 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":81583,"threads_started":1}
W20250626 19:58:46.132947 8221 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 19:58:46.057945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.46.188:38919 (local address 127.4.46.129:59495)
0626 19:58:46.131994 (+ 74049us) negotiation.cc:107] Waiting for socket to connect
0626 19:58:46.132202 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":73472,"thread_start_us":66647,"threads_started":1}
I20250626 19:58:46.155565 8154 consensus_queue.cc:1035] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 19:58:46.164263 8223 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 19:58:46.073057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.46.188:38919 (local address 127.0.0.1:60720)
0626 19:58:46.161883 (+ 88826us) negotiation.cc:107] Waiting for socket to connect
0626 19:58:46.162136 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":84190,"thread_start_us":90059,"threads_started":1}
W20250626 19:58:46.165284 6705 leader_election.cc:336] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6610553ba0df459085f7373935774377 (127.4.46.188:38919): Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection reset by peer (error 104)
I20250626 19:58:46.165655 6705 leader_election.cc:304] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 188d02ec81e645229acdc5eed98c9f9f; no voters: 6610553ba0df459085f7373935774377, 8ae38ec946e14198ae4adc8dde9d66e8
I20250626 19:58:46.166349 8167 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 19:58:46.180346 7759 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8ae38ec946e14198ae4adc8dde9d66e8 -> Peer 6610553ba0df459085f7373935774377 (127.4.46.188:38919): Couldn't send request to peer 6610553ba0df459085f7373935774377. Status: Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 19:58:46.190397 8154 consensus_queue.cc:1035] T 0084d8b5ae1548af8c943f8ace9bceb2 P bbe151be01e24c82854d9c8c79c797ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 19:58:46.192176 7787 catalog_manager.cc:5674] TS bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129) does not have the latest schema for tablet 0aaf2ab9ed7840898ca54cca6983eb82 (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49]). Expected version 1 got 0
@ 0x7f65cb745d82 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20250626 19:58:46.238373 8233 tablet.cc:1722] T 0aaf2ab9ed7840898ca54cca6983eb82 P bbe151be01e24c82854d9c8c79c797ca: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 19:58:46.251257 8232 tablet.cc:1722] T 0aaf2ab9ed7840898ca54cca6983eb82 P 297d2c913667408bae82668a557e06c7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250626 19:58:46.258505 7009 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.131:33703, user_credentials={real_user=slave}} blocked reactor thread for 80115.2us
I20250626 19:58:46.275161 7407 heartbeater.cc:499] Master 127.4.46.189:36417 was elected leader, sending a full tablet report...
W20250626 19:58:46.301545 7771 catalog_manager.cc:5415] Ignoring report from unknown tablet e267bd75d7d342a3a6c28b17eb37d83e
W20250626 19:58:46.301846 7771 catalog_manager.cc:5415] Ignoring report from unknown tablet 101f870043e44be7b3bf098c37b108af
W20250626 19:58:46.302210 7771 catalog_manager.cc:5415] Ignoring report from unknown tablet 734d9d3331b44fc49e1f64b8caf1e4d8
@ 0x7f65cb74507e std::__1::__function::__func<>::operator()() at ??:0
@ 0x59bcd5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x59bb0a std::__1::function<>::operator()() at ??:?
I20250626 19:58:46.338203 8236 tablet.cc:1722] T 0aaf2ab9ed7840898ca54cca6983eb82 P 315d304854c541bf8b5e6b1eea712d4a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
@ 0x7f65c2a82637 kudu::Thread::SuperviseThread() at ??:0
@ 0x465e60 __tsan_thread_start_func at ??:?
@ 0x7f65c41016db start_thread at ??:0
@ 0x7f65bf50c71f clone at ??:0
W20250626 19:58:46.734786 7151 connection.cc:537] client connection to 127.4.46.189:36417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250626 19:58:46.838228 8247 raft_consensus.cc:491] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8ae38ec946e14198ae4adc8dde9d66e8)
I20250626 19:58:46.838577 8247 raft_consensus.cc:513] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:46.840364 8247 leader_election.cc:290] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
W20250626 19:58:46.846392 6704 leader_election.cc:336] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417): Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
I20250626 19:58:46.847252 8235 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 19:58:46.847990 6705 leader_election.cc:336] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6610553ba0df459085f7373935774377 (127.4.46.188:38919): Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
I20250626 19:58:46.847767 8235 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:46.848340 6705 leader_election.cc:304] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 188d02ec81e645229acdc5eed98c9f9f; no voters: 6610553ba0df459085f7373935774377, 8ae38ec946e14198ae4adc8dde9d66e8
I20250626 19:58:46.848960 8247 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 19:58:46.849700 8235 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 19:58:46.851783 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 19:58:46.852135 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 19:58:46.852411 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:46.852942 8235 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:58:47.426091 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 6: this message will repeat every 5th retry.
I20250626 19:58:47.515463 8255 raft_consensus.cc:491] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8ae38ec946e14198ae4adc8dde9d66e8)
I20250626 19:58:47.516062 8255 raft_consensus.cc:513] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:47.517848 8255 leader_election.cc:290] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
W20250626 19:58:47.539671 6705 leader_election.cc:336] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6610553ba0df459085f7373935774377 (127.4.46.188:38919): Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
W20250626 19:58:47.540696 6704 leader_election.cc:336] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417): Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
I20250626 19:58:47.541049 6704 leader_election.cc:304] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 188d02ec81e645229acdc5eed98c9f9f; no voters: 6610553ba0df459085f7373935774377, 8ae38ec946e14198ae4adc8dde9d66e8
I20250626 19:58:47.541693 8255 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 19:58:49.111379 8265 raft_consensus.cc:491] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8ae38ec946e14198ae4adc8dde9d66e8)
I20250626 19:58:49.112342 8265 raft_consensus.cc:513] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:49.113855 8265 leader_election.cc:290] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
W20250626 19:58:49.119475 6704 leader_election.cc:336] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417): Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 19:58:49.120009 6705 leader_election.cc:336] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6610553ba0df459085f7373935774377 (127.4.46.188:38919): Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
I20250626 19:58:49.120375 6705 leader_election.cc:304] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 188d02ec81e645229acdc5eed98c9f9f; no voters: 6610553ba0df459085f7373935774377, 8ae38ec946e14198ae4adc8dde9d66e8
I20250626 19:58:49.120999 8265 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 19:58:50.845958 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 6: this message will repeat every 5th retry.
I20250626 19:58:51.194144 8281 raft_consensus.cc:491] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8ae38ec946e14198ae4adc8dde9d66e8)
I20250626 19:58:51.194643 8281 raft_consensus.cc:513] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188d02ec81e645229acdc5eed98c9f9f" member_type: VOTER last_known_addr { host: "127.4.46.190" port: 39311 } } peers { permanent_uuid: "8ae38ec946e14198ae4adc8dde9d66e8" member_type: VOTER last_known_addr { host: "127.4.46.189" port: 36417 } } peers { permanent_uuid: "6610553ba0df459085f7373935774377" member_type: VOTER last_known_addr { host: "127.4.46.188" port: 38919 } }
I20250626 19:58:51.200428 8281 leader_election.cc:290] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417), 6610553ba0df459085f7373935774377 (127.4.46.188:38919)
W20250626 19:58:51.200853 6704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20250626 19:58:51.204568 6704 leader_election.cc:336] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8ae38ec946e14198ae4adc8dde9d66e8 (127.4.46.189:36417): Network error: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 19:58:51.205233 6705 leader_election.cc:336] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6610553ba0df459085f7373935774377 (127.4.46.188:38919): Network error: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
I20250626 19:58:51.205601 6705 leader_election.cc:304] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 188d02ec81e645229acdc5eed98c9f9f; no voters: 6610553ba0df459085f7373935774377, 8ae38ec946e14198ae4adc8dde9d66e8
I20250626 19:58:51.206224 8281 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 188d02ec81e645229acdc5eed98c9f9f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 19:58:52.784233 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 6: this message will repeat every 5th retry.
W20250626 19:58:53.066896 6704 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e267bd75d7d342a3a6c28b17eb37d83e (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e267bd75d7d342a3a6c28b17eb37d83e consensus state has no leader
I20250626 19:58:55.904641 8335 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:58:55.905272 8335 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:58:55.907318 8335 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 19:58:55.908722 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 19:58:55.909427 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 19:58:55.909790 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:58:55.910441 8335 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:58:57.976084 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 11: this message will repeat every 5th retry.
W20250626 19:59:00.672685 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 11: this message will repeat every 5th retry.
W20250626 19:59:03.212199 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 11: this message will repeat every 5th retry.
W20250626 19:59:07.586071 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 16: this message will repeat every 5th retry.
W20250626 19:59:09.455535 6704 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e267bd75d7d342a3a6c28b17eb37d83e (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e267bd75d7d342a3a6c28b17eb37d83e consensus state has no leader
W20250626 19:59:10.297231 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 16: this message will repeat every 5th retry.
W20250626 19:59:13.050026 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 16: this message will repeat every 5th retry.
I20250626 19:59:15.647157 8495 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:59:15.647703 8495 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:59:15.650399 8495 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 19:59:15.651926 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 19:59:15.652509 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 19:59:15.652801 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:59:15.653461 8495 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:59:16.686546 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 21: this message will repeat every 5th retry.
W20250626 19:59:20.606171 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 21: this message will repeat every 5th retry.
W20250626 19:59:22.801371 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 21: this message will repeat every 5th retry.
W20250626 19:59:26.078622 7407 heartbeater.cc:646] Failed to heartbeat to 127.4.46.189:36417 (78 consecutive failures): Network error: Failed to ping master at 127.4.46.189:36417: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 19:59:26.303851 7261 heartbeater.cc:646] Failed to heartbeat to 127.4.46.188:38919 (161 consecutive failures): Network error: Failed to ping master at 127.4.46.188:38919: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
I20250626 19:59:26.505991 8495 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:59:26.506574 8495 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:59:26.508742 8495 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 19:59:26.509938 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 19:59:26.510491 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 19:59:26.510843 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:59:26.511451 8495 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:59:26.513036 7119 heartbeater.cc:646] Failed to heartbeat to 127.4.46.189:36417 (79 consecutive failures): Network error: Failed to ping master at 127.4.46.189:36417: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 19:59:26.633880 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 26: this message will repeat every 5th retry.
W20250626 19:59:30.819118 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 26: this message will repeat every 5th retry.
W20250626 19:59:32.205593 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 26: this message will repeat every 5th retry.
W20250626 19:59:36.512635 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 31: this message will repeat every 5th retry.
W20250626 19:59:40.042436 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 31: this message will repeat every 5th retry.
W20250626 19:59:42.044718 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 31: this message will repeat every 5th retry.
W20250626 19:59:42.244597 6704 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e267bd75d7d342a3a6c28b17eb37d83e (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e267bd75d7d342a3a6c28b17eb37d83e consensus state has no leader
I20250626 19:59:44.565789 8700 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 19:59:44.566315 8700 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 19:59:44.568415 8700 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 19:59:44.569983 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 19:59:44.570552 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 19:59:44.570889 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 19:59:44.571584 8700 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 19:59:46.984426 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 36: this message will repeat every 5th retry.
W20250626 19:59:50.363139 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 36: this message will repeat every 5th retry.
W20250626 19:59:51.433384 6749 debug-util.cc:398] Leaking SignalData structure 0x7b0800095520 after lost signal to thread 6686
W20250626 19:59:51.434150 6749 debug-util.cc:398] Leaking SignalData structure 0x7b080004ef20 after lost signal to thread 6752
W20250626 19:59:52.320418 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 36: this message will repeat every 5th retry.
W20250626 19:59:57.083114 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 41: this message will repeat every 5th retry.
W20250626 19:59:57.731195 6749 debug-util.cc:398] Leaking SignalData structure 0x7b08000b3ee0 after lost signal to thread 6686
W20250626 19:59:57.731798 6749 debug-util.cc:398] Leaking SignalData structure 0x7b08000ae860 after lost signal to thread 6752
W20250626 20:00:00.532366 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 41: this message will repeat every 5th retry.
W20250626 20:00:01.651890 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 41: this message will repeat every 5th retry.
I20250626 20:00:04.478698 8879 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:00:04.479308 8879 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:00:04.481158 8879 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:00:04.482779 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:00:04.483592 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:00:04.483908 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:00:04.484516 8879 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:00:07.437923 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 46: this message will repeat every 5th retry.
W20250626 20:00:08.229473 7402 debug-util.cc:398] Leaking SignalData structure 0x7b080003e9e0 after lost signal to thread 7273
W20250626 20:00:08.230778 7402 debug-util.cc:398] Leaking SignalData structure 0x7b080010d580 after lost signal to thread 7405
W20250626 20:00:10.953454 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 46: this message will repeat every 5th retry.
W20250626 20:00:11.744860 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08001176a0 after lost signal to thread 7129
W20250626 20:00:11.746289 7257 debug-util.cc:398] Leaking SignalData structure 0x7b0800122c60 after lost signal to thread 7260
W20250626 20:00:11.906925 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 46: this message will repeat every 5th retry.
I20250626 20:00:17.068410 6707 maintenance_manager.cc:419] P 188d02ec81e645229acdc5eed98c9f9f: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033369
I20250626 20:00:17.157390 6706 maintenance_manager.cc:643] P 188d02ec81e645229acdc5eed98c9f9f: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.088s user 0.066s sys 0.018s Metrics: {"bytes_written":37574,"cfile_init":1,"drs_written":1,"lbm_read_time_us":306,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2345,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":96,"thread_start_us":662,"threads_started":1,"wal-append.queue_time_us":962}
I20250626 20:00:17.159246 6707 maintenance_manager.cc:419] P 188d02ec81e645229acdc5eed98c9f9f: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7069 bytes on disk
I20250626 20:00:17.160533 6706 maintenance_manager.cc:643] P 188d02ec81e645229acdc5eed98c9f9f: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":163,"lbm_reads_lt_1ms":4}
W20250626 20:00:17.551558 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 51: this message will repeat every 5th retry.
W20250626 20:00:18.798964 7114 debug-util.cc:398] Leaking SignalData structure 0x7b0800102aa0 after lost signal to thread 6982
W20250626 20:00:18.800189 7114 debug-util.cc:398] Leaking SignalData structure 0x7b08000eb0c0 after lost signal to thread 7117
W20250626 20:00:21.355453 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 51: this message will repeat every 5th retry.
W20250626 20:00:22.213454 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 51: this message will repeat every 5th retry.
I20250626 20:00:23.802829 9005 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:00:23.803346 9005 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:00:23.805351 9005 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:00:23.807361 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:00:23.808125 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:00:23.808709 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:00:23.809437 9005 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:00:26.445495 7407 heartbeater.cc:646] Failed to heartbeat to 127.4.46.189:36417 (198 consecutive failures): Network error: Failed to ping master at 127.4.46.189:36417: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 20:00:26.588097 7262 heartbeater.cc:646] Failed to heartbeat to 127.4.46.189:36417 (385 consecutive failures): Network error: Failed to ping master at 127.4.46.189:36417: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 20:00:26.917363 7119 heartbeater.cc:646] Failed to heartbeat to 127.4.46.189:36417 (199 consecutive failures): Network error: Failed to ping master at 127.4.46.189:36417: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 20:00:27.280332 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 56: this message will repeat every 5th retry.
I20250626 20:00:27.776526 9005 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:00:27.776911 9005 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:00:27.778625 9005 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:00:27.780086 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:00:27.780506 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:00:27.780911 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:00:27.781404 9005 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:00:30.725615 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 56: this message will repeat every 5th retry.
W20250626 20:00:31.393157 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 56: this message will repeat every 5th retry.
W20250626 20:00:35.125649 7402 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea4e0 after lost signal to thread 7273
W20250626 20:00:35.126946 7402 debug-util.cc:398] Leaking SignalData structure 0x7b08000d79a0 after lost signal to thread 7405
I20250626 20:00:35.603091 9005 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:00:35.603610 9005 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:00:35.605921 9005 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:00:35.607990 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:00:35.608552 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:00:35.608909 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:00:35.609697 9087 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:00:36.510243 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 61: this message will repeat every 5th retry.
I20250626 20:00:39.027967 9099 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:00:39.028396 9099 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
W20250626 20:00:39.034075 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:00:39.030498 9099 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:00:39.060943 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:00:39.074393 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:00:39.075151 9099 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:00:40.074544 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08000d8260 after lost signal to thread 7129
W20250626 20:00:40.075449 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08000d8080 after lost signal to thread 7260
W20250626 20:00:40.326315 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 61: this message will repeat every 5th retry.
W20250626 20:00:41.513327 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 61: this message will repeat every 5th retry.
W20250626 20:00:42.254452 6704 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e267bd75d7d342a3a6c28b17eb37d83e (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e267bd75d7d342a3a6c28b17eb37d83e consensus state has no leader
W20250626 20:00:46.642577 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 66: this message will repeat every 5th retry.
I20250626 20:00:48.492756 9184 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:00:48.493350 9184 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:00:48.495357 9184 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:00:48.496490 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:00:48.497063 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:00:48.497438 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:00:48.498085 9184 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:00:50.395529 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 66: this message will repeat every 5th retry.
W20250626 20:00:51.122135 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 66: this message will repeat every 5th retry.
I20250626 20:00:52.700402 9184 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:00:52.700912 9184 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:00:52.702566 9184 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:00:52.704159 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:00:52.704630 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:00:52.705011 7006 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:00:52.705657 9184 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:00:56.390399 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 71: this message will repeat every 5th retry.
W20250626 20:00:57.291143 7114 debug-util.cc:398] Leaking SignalData structure 0x7b08000db380 after lost signal to thread 6982
W20250626 20:00:57.292054 7114 debug-util.cc:398] Leaking SignalData structure 0x7b08000f9c40 after lost signal to thread 7117
W20250626 20:00:57.961061 6749 debug-util.cc:398] Leaking SignalData structure 0x7b080003eca0 after lost signal to thread 6686
W20250626 20:00:57.961773 6749 debug-util.cc:398] Leaking SignalData structure 0x7b080004c080 after lost signal to thread 6752
W20250626 20:01:00.450214 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 71: this message will repeat every 5th retry.
W20250626 20:01:00.625212 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 71: this message will repeat every 5th retry.
I20250626 20:01:05.862849 9350 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:01:05.863513 9350 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:01:05.865967 9350 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:01:05.867806 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:01:05.868248 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:01:05.868592 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:01:05.869208 9350 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:01:06.907671 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 76: this message will repeat every 5th retry.
W20250626 20:01:10.124585 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 76: this message will repeat every 5th retry.
W20250626 20:01:10.785243 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 76: this message will repeat every 5th retry.
W20250626 20:01:13.647578 7114 debug-util.cc:398] Leaking SignalData structure 0x7b08000f8540 after lost signal to thread 6982
W20250626 20:01:13.648492 7114 debug-util.cc:398] Leaking SignalData structure 0x7b08000dc560 after lost signal to thread 7117
I20250626 20:01:16.329612 9403 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:01:16.330092 9403 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:01:16.331717 9403 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:01:16.333184 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:01:16.333684 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:01:16.333942 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:01:16.334868 9403 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:01:17.424327 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 81: this message will repeat every 5th retry.
W20250626 20:01:19.481094 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 81: this message will repeat every 5th retry.
W20250626 20:01:21.235126 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 81: this message will repeat every 5th retry.
W20250626 20:01:25.320561 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08000f3a60 after lost signal to thread 7129
W20250626 20:01:25.321530 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08000ffa80 after lost signal to thread 7260
W20250626 20:01:26.617980 7408 heartbeater.cc:646] Failed to heartbeat to 127.4.46.190:39311 (106 consecutive failures): Timed out: Failed to ping master at 127.4.46.190:39311: Ping RPC to 127.4.46.190:39311 timed out after 1.000s (SENT)
W20250626 20:01:26.950173 7261 heartbeater.cc:646] Failed to heartbeat to 127.4.46.188:38919 (620 consecutive failures): Network error: Failed to ping master at 127.4.46.188:38919: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
W20250626 20:01:27.323338 7119 heartbeater.cc:646] Failed to heartbeat to 127.4.46.189:36417 (319 consecutive failures): Network error: Failed to ping master at 127.4.46.189:36417: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 20:01:27.456000 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 86: this message will repeat every 5th retry.
W20250626 20:01:29.730077 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 86: this message will repeat every 5th retry.
W20250626 20:01:31.584175 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 86: this message will repeat every 5th retry.
I20250626 20:01:35.508409 9503 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:01:35.508944 9503 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:01:35.511240 9503 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:01:35.512580 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:01:35.513105 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:01:35.513463 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:01:35.514022 9503 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:01:38.631575 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 91: this message will repeat every 5th retry.
W20250626 20:01:39.321996 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 91: this message will repeat every 5th retry.
W20250626 20:01:41.470306 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 91: this message will repeat every 5th retry.
W20250626 20:01:41.575604 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08000c2680 after lost signal to thread 7129
W20250626 20:01:41.576428 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08000f23a0 after lost signal to thread 7260
W20250626 20:01:42.295509 6704 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e267bd75d7d342a3a6c28b17eb37d83e (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e267bd75d7d342a3a6c28b17eb37d83e consensus state has no leader
I20250626 20:01:42.769284 9679 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:01:42.769893 9679 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:01:42.771735 9679 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:01:42.773063 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:01:42.773646 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:01:42.773998 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:01:42.774616 9679 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:01:47.961788 7402 debug-util.cc:398] Leaking SignalData structure 0x7b08000b19c0 after lost signal to thread 7273
W20250626 20:01:47.962877 7402 debug-util.cc:398] Leaking SignalData structure 0x7b0800108a80 after lost signal to thread 7405
W20250626 20:01:49.066242 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 96: this message will repeat every 5th retry.
W20250626 20:01:49.072423 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 96: this message will repeat every 5th retry.
W20250626 20:01:51.981588 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 96: this message will repeat every 5th retry.
I20250626 20:01:55.753295 9743 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:01:55.753751 9743 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:01:55.755659 9743 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:01:55.757066 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:01:55.757536 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:01:55.757939 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:01:55.758600 9743 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:01:59.321866 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 101: this message will repeat every 5th retry.
W20250626 20:01:59.440459 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 101: this message will repeat every 5th retry.
W20250626 20:02:01.842694 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 101: this message will repeat every 5th retry.
W20250626 20:02:08.932953 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08001229c0 after lost signal to thread 7129
W20250626 20:02:08.933820 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea360 after lost signal to thread 7260
I20250626 20:02:09.234153 9967 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:02:09.234643 9967 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:02:09.236265 9967 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:02:09.237995 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:02:09.238543 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:02:09.238827 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:02:09.239320 9967 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:02:09.735080 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 106: this message will repeat every 5th retry.
W20250626 20:02:09.749950 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 106: this message will repeat every 5th retry.
W20250626 20:02:11.634732 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 106: this message will repeat every 5th retry.
I20250626 20:02:17.536485 10056 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:02:17.537050 10056 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:02:17.539064 10056 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:02:17.540607 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:02:17.541098 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:02:17.541381 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:02:17.542045 10056 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:02:19.204102 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 111: this message will repeat every 5th retry.
W20250626 20:02:19.607546 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 111: this message will repeat every 5th retry.
W20250626 20:02:21.612680 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 111: this message will repeat every 5th retry.
W20250626 20:02:26.705489 7408 heartbeater.cc:646] Failed to heartbeat to 127.4.46.190:39311 (146 consecutive failures): Timed out: Failed to ping master at 127.4.46.190:39311: Ping RPC to 127.4.46.190:39311 timed out after 1.000s (SENT)
W20250626 20:02:27.010291 7262 heartbeater.cc:646] Failed to heartbeat to 127.4.46.189:36417 (845 consecutive failures): Network error: Failed to ping master at 127.4.46.189:36417: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 20:02:27.687915 7118 heartbeater.cc:646] Failed to heartbeat to 127.4.46.188:38919 (444 consecutive failures): Network error: Failed to ping master at 127.4.46.188:38919: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
W20250626 20:02:29.975809 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 116: this message will repeat every 5th retry.
W20250626 20:02:30.196712 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 116: this message will repeat every 5th retry.
I20250626 20:02:31.202172 10202 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:02:31.202718 10202 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:02:31.204795 10202 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:02:31.207031 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:02:31.207572 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:02:31.207854 7006 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:02:31.208425 10202 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:02:31.910660 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 116: this message will repeat every 5th retry.
I20250626 20:02:39.074007 10202 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:02:39.074492 10202 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:02:39.076666 10202 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:02:39.078050 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:02:39.078454 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:02:39.078718 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:02:39.079214 10202 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:02:39.409576 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 121: this message will repeat every 5th retry.
W20250626 20:02:39.823491 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 121: this message will repeat every 5th retry.
W20250626 20:02:42.342829 6704 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e267bd75d7d342a3a6c28b17eb37d83e (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e267bd75d7d342a3a6c28b17eb37d83e consensus state has no leader
W20250626 20:02:42.578902 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 121: this message will repeat every 5th retry.
W20250626 20:02:44.652571 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08000be040 after lost signal to thread 7129
W20250626 20:02:44.653416 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08000416c0 after lost signal to thread 7260
I20250626 20:02:48.868309 10316 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:02:48.868757 10316 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:02:48.870656 10316 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:02:48.871914 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:02:48.872431 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:02:48.872684 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:02:48.873132 10316 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:02:49.521164 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 126: this message will repeat every 5th retry.
W20250626 20:02:49.918495 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 126: this message will repeat every 5th retry.
W20250626 20:02:52.546828 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 126: this message will repeat every 5th retry.
I20250626 20:02:53.874228 10316 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:02:53.874859 10316 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:02:53.877233 10316 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:02:53.878453 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:02:53.878971 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:02:53.879305 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:02:53.879879 10316 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:02:56.767344 6749 debug-util.cc:398] Leaking SignalData structure 0x7b080003d6a0 after lost signal to thread 6686
W20250626 20:02:56.767865 6749 debug-util.cc:398] Leaking SignalData structure 0x7b0800049360 after lost signal to thread 6752
W20250626 20:02:59.089334 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 131: this message will repeat every 5th retry.
W20250626 20:02:59.784155 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 131: this message will repeat every 5th retry.
W20250626 20:03:02.820436 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 131: this message will repeat every 5th retry.
W20250626 20:03:06.807931 7114 debug-util.cc:398] Leaking SignalData structure 0x7b08000da500 after lost signal to thread 6982
W20250626 20:03:06.809069 7114 debug-util.cc:398] Leaking SignalData structure 0x7b0800116de0 after lost signal to thread 7117
W20250626 20:03:09.233264 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 136: this message will repeat every 5th retry.
W20250626 20:03:10.230890 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 136: this message will repeat every 5th retry.
W20250626 20:03:12.439595 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 136: this message will repeat every 5th retry.
I20250626 20:03:13.078922 10496 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:03:13.079604 10496 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:03:13.082079 10496 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:03:13.083482 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:03:13.083957 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:03:13.084177 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:03:13.084702 10496 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:03:19.438344 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 141: this message will repeat every 5th retry.
W20250626 20:03:20.892184 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 141: this message will repeat every 5th retry.
W20250626 20:03:22.211709 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 141: this message will repeat every 5th retry.
W20250626 20:03:26.037652 7402 debug-util.cc:398] Leaking SignalData structure 0x7b0800108a20 after lost signal to thread 7273
W20250626 20:03:26.038513 7402 debug-util.cc:398] Leaking SignalData structure 0x7b0800110280 after lost signal to thread 7405
W20250626 20:03:26.787698 7408 heartbeater.cc:646] Failed to heartbeat to 127.4.46.190:39311 (186 consecutive failures): Timed out: Failed to ping master at 127.4.46.190:39311: Ping RPC to 127.4.46.190:39311 timed out after 1.000s (SENT)
W20250626 20:03:27.120262 7262 heartbeater.cc:646] Failed to heartbeat to 127.4.46.189:36417 (1071 consecutive failures): Network error: Failed to ping master at 127.4.46.189:36417: Client connection negotiation failed: client connection to 127.4.46.189:36417: connect: Connection refused (error 111)
W20250626 20:03:27.320850 7294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.46.129:33129, user_credentials={real_user=slave}} blocked reactor thread for 107333us
W20250626 20:03:27.998486 7118 heartbeater.cc:646] Failed to heartbeat to 127.4.46.188:38919 (564 consecutive failures): Network error: Failed to ping master at 127.4.46.188:38919: Client connection negotiation failed: client connection to 127.4.46.188:38919: connect: Connection refused (error 111)
W20250626 20:03:28.318611 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08000b57e0 after lost signal to thread 7129
W20250626 20:03:28.319617 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08000f5ca0 after lost signal to thread 7260
I20250626 20:03:29.286192 10681 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:03:29.286675 10681 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:03:29.289057 10681 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:03:29.290390 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:03:29.290961 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:03:29.291322 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:03:29.291818 10681 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:03:30.076236 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 146: this message will repeat every 5th retry.
W20250626 20:03:30.270018 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 146: this message will repeat every 5th retry.
W20250626 20:03:32.043315 7152 consensus_peers.cc:487] T 089e49e29b2b4f3dbea8a4597d9e36a2 P 315d304854c541bf8b5e6b1eea712d4a -> Peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Couldn't send request to peer 297d2c913667408bae82668a557e06c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 089e49e29b2b4f3dbea8a4597d9e36a2. This is attempt 146: this message will repeat every 5th retry.
W20250626 20:03:34.915166 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08001097c0 after lost signal to thread 7129
W20250626 20:03:34.916205 7257 debug-util.cc:398] Leaking SignalData structure 0x7b08000dae20 after lost signal to thread 7260
I20250626 20:03:38.817371 10681 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:03:38.817811 10681 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:03:38.819420 10681 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:03:38.820811 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:03:38.821527 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:03:38.821849 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:03:38.822561 10818 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:03:39.781024 7150 consensus_peers.cc:487] T 101f870043e44be7b3bf098c37b108af P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101f870043e44be7b3bf098c37b108af. This is attempt 151: this message will repeat every 5th retry.
W20250626 20:03:40.806721 7150 consensus_peers.cc:487] T e267bd75d7d342a3a6c28b17eb37d83e P 315d304854c541bf8b5e6b1eea712d4a -> Peer bbe151be01e24c82854d9c8c79c797ca (127.4.46.129:33129): Couldn't send request to peer bbe151be01e24c82854d9c8c79c797ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e267bd75d7d342a3a6c28b17eb37d83e. This is attempt 151: this message will repeat every 5th retry.
I20250626 20:03:42.012277 10852 raft_consensus.cc:491] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:03:42.012725 10852 raft_consensus.cc:513] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe151be01e24c82854d9c8c79c797ca" member_type: VOTER last_known_addr { host: "127.4.46.129" port: 33129 } } peers { permanent_uuid: "315d304854c541bf8b5e6b1eea712d4a" member_type: VOTER last_known_addr { host: "127.4.46.130" port: 40373 } } peers { permanent_uuid: "297d2c913667408bae82668a557e06c7" member_type: VOTER last_known_addr { host: "127.4.46.131" port: 33703 } }
I20250626 20:03:42.014258 10852 leader_election.cc:290] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373), 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703)
W20250626 20:03:42.015364 7006 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315d304854c541bf8b5e6b1eea712d4a (127.4.46.130:40373): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
W20250626 20:03:42.015820 7009 leader_election.cc:343] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 297d2c913667408bae82668a557e06c7 (127.4.46.131:33703): Not found: Tablet not found: ae04997b77924e8ba479c5c34ac822d7
I20250626 20:03:42.016067 7009 leader_election.cc:304] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe151be01e24c82854d9c8c79c797ca; no voters: 297d2c913667408bae82668a557e06c7, 315d304854c541bf8b5e6b1eea712d4a
I20250626 20:03:42.016556 10852 raft_consensus.cc:2747] T ae04997b77924e8ba479c5c34ac822d7 P bbe151be01e24c82854d9c8c79c797ca [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 20:03:42.361924 6704 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e267bd75d7d342a3a6c28b17eb37d83e (table default.table_4e311e30e4244e30aa0506381ffc07c0 [id=8768a4a6b5824e7db4f71bd958c17f49], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e267bd75d7d342a3a6c28b17eb37d83e consensus state has no leader
W20250626 20:03:42.401386 7418 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToMaster RPC to 127.4.46.190:39311 timed out after 55.631s (SENT)
W20250626 20:03:42.402055 7418 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToMaster RPC to 127.4.46.190:39311 timed out after 55.631s (SENT)
W20250626 20:03:42.402289 7418 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToMaster RPC to 127.4.46.190:39311 timed out after 55.631s (SENT)
F20250626 20:03:42.413740 7429 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToMaster RPC to 127.4.46.190:39311 timed out after 55.631s (SENT)
*** Check failure stack trace: ***
*** Aborted at 1750968222 (unix time) try "date -d @1750968222" if you are using GNU date ***
F20250626 20:03:42.419513 7427 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_afaa8d56134e47008a0eb5b4de5c35da on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToMaster RPC to 127.4.46.190:39311 timed out after 55.631s (SENT)
PC: @ 0x0 (unknown)
*** Check failure stack trace: ***
*** SIGABRT (@0x3e8000010ba) received by PID 4282 (TID 0x7f11d07fa700) from PID 4282; stack trace: ***
@ 0x2dae00 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2dd2e3 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f11e401d980 (unknown) at ??:0
@ 0x7f11df25dfb7 gsignal at ??:0
@ 0x7f11df25f921 abort at ??:0
@ 0x2e0df6 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f11e22f5da4 google::LogMessage::SendToLog() at ??:0
@ 0x7f11e22f6950 google::LogMessage::Flush() at ??:0
@ 0x7f11e22fba8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x35d526 kudu::MasterStressTest::AlterTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38ae0e _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3846
@ 0x38acc1 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x38a40b std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2dacc0 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7f11e40126db start_thread at ??:0
@ 0x7f11df34071f clone at ??:0